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IS  ABSTRACT 


A  study  was  performed  to  evaluate  the  training  effectiveness  of  Communication  and 
Navigation  Training  Device  1D23.  Foui  objectives  were  addressed:  Assess  impacts  of 
training  in  the  device  upon  student  dead  reckoning  navigation  performance.  Determine 
impacts  of  trainir.g  in  the  device  upon  communication  and  navigation  performance  in 
point-to-point  and  airways  navigation.  Determine  impacts  of  increased  amounts  of 
training  in  the  device  upon  communication  and  navigation  performance  in  dead  reckoning^ 
point-to-point,  and  airways  navigation.  Develop  design  oriented  information  for 
application  to  further  training  device  designs.  Flight  grades  and  supplemental 
measures  of  student  performance  ere  analyzed.  Instructor  and  student  opinion  data 
were  obtained  to  supplement  direct  observations  by  the  author.  Training  effectiveness 
of  the  device  for  dead  reckoning,  point-to-point  and  airways  navigation  was  documented. 
No  performance  improvements  were  found  as  a  result  of  the  additional  training  which 
was  examined.  Design  oriented  information  was  developed. 
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FOREWORD 


The  Naval  Training  Equipment  Center  has  a  continuing  interest  in  the 
evaluation  of  training  to  improve  training  device  utilization,  to  develop 
information  for  use  in  future  training  device  design,  and  last  but  not  least, 
to  improve  the  methodology  and  conduct  of  training  evaluations.  These  goals 
were  taken  as  objectives  for  the  present  efrort. 

With  respect  to  training  device  utilization,  the  present  study  in¬ 
vestigated  the  relation  ship  between  training  in  Device  1D23,  a  new  trainer, 
and  subsequent  performance  in  the  airborne  training  phases.  With  respect 
to  training  device  design,  the  present  study  evaluated  the  adequacy  of 
certain  advanced  design  features  of  Device  1D23  such  as  the  generalized 
trainee  station  and  automatic  performance  measurement  capability.  In  addi¬ 
tion,  the  use  of  current  grading  practices  for  evaluation  purposes  was  also 
investigated.  The  experimental  data  provide  evidence  that  current  gra<  !ng 
practices  are  not  sensitive  to  changes  in  navigation  performance  and, 
perhaps,  more  important,  that  the  implementation  of  certain  experimental 
design  conditions  (namely  control  group  data  and  objective  performance 
measurement)  are  crucial  to  successful  transfer-of-training  evaluations. 
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SECTION  I 
INTRODUCTION 


This  report  documents  a  training  effectiveness  evaluation 
of  Device  1D23,  Communication  and  Navigation  Trainer. 

Purposes  of  the  evaluation  were  to  provide  objective  bases 
for  assessing  the  training  effectiveness  of  the  device 
relative  to  its  use  in  a  specific  syllabus  of  instruction, 
and  to  provide  guidelines  for  future  evaluations  and  training 
device  desians.  The  evaluation  centered  uoon  the  collection 
and  analysis  of  student  performance  data  and  other 
information  in  a  formal  and  systematic  manner. 

DEVICE  CHARACTERISTICS 

Device  1D23  was  a  fixed  base,  digitally  driven  trainer 
which  was  designed  to  orovide  training  in  the  cognitive 
aspects  of  basic  aircraft  navigation  and  practice  in  radio 
communication  required  in  the  operation  of  typical  Navy 
aircraft  such  as  the  F-4J  and  the  E-2A.  It  is  described  more 
fully  in  Reference  1. 

The  device  recently  was  installed  at  Training  Squadron 
Ten  (VT-10) ,  Naval  Air  Station,  Pensacola,  Florida,  where 
it  was  used  in  basic  Naval  Flight  Officer  (NFO)  training. 
During  the  evaluation  it  was  used  to  provide  training  in 
the  following: 

Navigation  communication  techniques 
Basic  dead  reckoning 
Airways  navigation 
Solv'.ng  relative  motion  problems 
Aircraft  fuel  management 
Integration  of  navigation  aids 

The  device  incorporated  40  identical  student  stations, 
each  of  which  generally  simulated  the  NFO  cockpit  of  a  typical 
Navy  aircraft.  All  stations  were  capable  of  independent 
operation  and  maneuvering  within  a  1200  nautical  mile  problem 
area. 


Each  student  station  contained  instruments,  controls 
and  displays  to  permit  the  student  to  communicate,  make 
navigation  inputs  to  the  system,  and  make  necessary  inputs 
to  the  digital  computer  which  drove  the  trainer.  The 
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computer  also  drove  instruments  and  displays  in  accordance 
with  each  student’s  inputs  as  he  attempted  to  navigate 
over  preplanned  mission  courses.  The  spectrum  of  training 
which  was  provided  in  the  device  is  reflected  by  the 
system  capabilities  which  were  available  at  each  student 
station.  Each  station  contained  the  following: 


Magnetic  compass 
Time  display 
Airspeed/mach  indicator 
Attitude  indicator 


Coramutticattion  .navigation 
identification  (CNI)  panel 

CNI  data  entry  panel 

Navigation  computer  display 
panel 


Pressure  altimeter 

Radar  altimeter 

Radio  magnetic  indicator 

Climb  rate  indicator 

Distance  measuring  equipment 

Navigational  mode  and 
data  entry  panel 

Audio  recorder 


Inertial  doppler  control 
panel 

Communications  headset 

Command  and  response  control 
panel  for  trainer  computer 
input 

Performance  alarm  panel 

General-purpose  alpha¬ 
numeric  display  panel 

Fuel  Flow  Indicator 


One  of  the  unique  features  of  the  trainer  w as  that  all  but 
three  displays  incorporated  light  emitting  diode  ligital  readouts. 
Only  the  magnetic  compass,  radio  magnetic  indicator,  and  the 
attitude  indicator  were  analog  displays. 


The  device  utilized  11  preprogrammed  training  problems 
(TPs)  which  progressively  varied  in  level  of  difficulty. 

Up  to  40  student  stations  could  be  designated  for  training  on 
any  one  of  the  TPs  at  any  given  time.  Training  on  two 
different  TPs  could  be  independently  undertaken  at  any 
qiven  time,  with  up  to  20  stations  dedicated  to  each  TP. 

The  TPs  provided  the  context  for  structured  training, 
student  cueing,  knowledge  of  results  (KOR)  feedback,  and 
automated  student  performance  evaluation.  For  automated 
performance  evaluation,  each  student's  performance  could 
be  independently  compared  with  that  of  an  "ideal  navigator." 
Student  stations  also  could  be  operated  independently  of 
the  TPs,  thus  providing  the  capability  for  unstructured 
practice.  The  TPs  are  summarized  in  Appendix  A.  A  more 
detailed  description  of  the  TPs  can  be  found  in  Reference  2. 
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Two  sets  of  Instructor  and  Training  Device  Operator  (TDO) 
consoles  contained  interactive  graphics  terminals  and  data 
entry  keyboards  for  controlling  the  trainer,  monitoring  and 
controlling  student  stations,  and  communicating  with 
student  stations.  During  a  typical  training  mission, 
one  nstructor  with  two  TDOs  were  required  for  each  block 
of  20  student  stations. 

UTILIZATION  CONTEXT 

At  the  time  of  the  evaluation  study,  basic  NFO  training 
required  a  minimum  of  20  weeks  at  VT-10.  Each  class 
consisted  of  an  average  of  36  students.  On  a  time  basis, 
training  consisted  of  approximately  60  percent  academics  (includ¬ 
ing  training  in  Device  1D23)  and  40  percent  in-flight  training. 
Before  being  assigned  to  a  replacement  air  group  (RAG) , 
graduates  of  VT-10  were  required  to  accomplish  an  additional 
six  to  13  weeks  of  advanced  training  at  other  schools, 
depending  upon  the  specialty  for  which  they  were  preparing. 

At  VT-10  each  student  received  approximately  eight 
weeks  of  academic  instruction  prior  to  receiving  training  in 
Device  1D23.  The  students'  first  formal  exposure  to  the 
device  involved  TPs  1  and  2,  which  were  designed  to  introduce 
students  to  the  operation  and  use  of  the  trainer.  Students 
then  received  training  on  TPs  three  through  six.  This 
training  was  followed  by  four  in-flight  dead  reckoning 
navigation  training  flights  in  T-29  or  C-114  aircraft. 

Following  approximately  two  weeks  of  additional  academic 
training,  students  received  training  on  TPs  7,  8,  and  10. 

This  was  followed  by  three  training  flights  in  T-39  aircraft. 
Point-to-point  and  airways  navigation  and  communication 
techniques  were  emphasised  during  thes;»  flights.  Following 
several  days  of  additional  academic  training,  students 
received  training  on  TPs  9,  10,  and  11.  This  training 
was  followed  by  in-flight  instruction  in  F-9  aircraft. 

In  addition  to  specified  training  in  the  device 
as  is  described  briefly  above,  the  device  was  kept  available 
approximately  four  hours  per  day,  during  which  time  students 
could  voluntarily  come  in  for  additional  training. 

Objectives  set  forth  for  the  evaluation  study  were 
accomplished  in  the  framework  of  device  characteristics  and 
utilization  context  as  summarized  above. 
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SECTION  II 

OBJECTIVES  AND  CONSTRAINTS 


OBJECTIVES 

Four  separate,  yet  related  objectives  were  set  forth  for 
the  training  effectiveness  evaluation  study.  They  were: 

Assess  impacts  of  training  in  Device  1D23  upon 
student  dead  reckoning  navigation  performance  as 
demonstrated  during  in-flight  training  in  the  T-29 
or  C-114  aircraft. 

Determine  impacts  of  training  in  Device  1D23  upon 
student  communication  and  navigation  task  perform¬ 
ance  in  the  T-39  airborne  training  phase  (point- 
to-point  and  airways  navigation) . 

Determine  impacts  of  increased  amounts  of  training 
in  Device  1D23  upon  student  communication  and 
navigation  task  performance  in  both  the  T-29/C-114 
and  the  T-39  airborne  training  phases. 

Develop  design  oriented  information  and  data 
for  application  to  future  training  device 
designs . 

Accomplishment  of  all  study  objectives  was  achieved 
within  the  context  of  several  training  situation  constraints 
which  are  discussed  below. 

CONSTRAINTS 

Every  evaluation  study  is  subject  to  certain  constraints. 
There  are  two  purposes  for  describing  the  constraints  which 
bounded  the  present  study.  The  first  is  to  define  the 
limiting  factors  which  impacted  upon  the  planning  and 
execution  of  the  study  for  the  purpose  of  qualifying  the 
results  of  the  study.  The  second  is  to  make  available 
information  which  may  be  of  practical  value  in  planning 
sr*ci  cxccUwin^  siniilsr  svsluitions , 

A  primary  constraining  factor  was  that  training  in 
Device  1D23  had  been  instituted  prior  to  detailed  planning 
and  initiation  of  the  evaluation  study.  This  general  factor 
resulted  in  several  specific  constraints,  each  of  which  is 
discussed  in  the  following  paragraphs. 
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Use  of  the  trainer  had  been  tightly  integrated  into 
NFO  training  at  VT-10.  Because  of  pipeline  demands  and  limited 
instructional  resources,  major  alterations  to  the  existing 
use  of  the  device  or  to  in-flight  training  practices  were 
judged  to  be  unacceptably  dir»tuptive  to  training. 

Accordingly,  the  creation  of  special  student  control  groups, 
which  would  have  advanced  to  in-flight  training  without 
training  in  the  device,  had  to  be  ruled  out.  Instructional 
personnel  also  were  strongly  opposed  to  special  control 
groups  on  the  basis  that  they  felt  the  trainer  was  improving 
student  performance  and  that  students  should  not  be  deprived 
of  this  perceived  benefit. 


The  fact  that  the  trainer  already  was  in  use  also 
cascaded  into  the  area  of  student  performance  measurement. 
Reliable,  task-oriented  measurement  is  a  virtual  necessity 
for  any  type  of  device  evaluation.  Obtaining  such  measurement, 
however,  frequently  requires  additional,  resources  which  may 
not  be  available  without  disrupting  training  or  causing 
pipeline  demands  to  go  unset. 

Instructional  resource  limitations  coupled  with  the  need 
to  maintain  existing  rates  of  student  flew  through  VT—10 
greatly  constrained  the  measurement  of  student  performance 
of  dead  reckoning  navigation  tasks.  Student  to 
instructor  ratios  of  eight  to  one  were  common  on  dead 
reckoning  training  flights.  At  a  minimum,  improved 
measurement  would  have  required  an  improved  student  to 
instructor  ratio,  as  well  as  increased  opportunities 
1n-fliqht  for  students  to  act  as  the  aircraft's  lead 
navigator.  Neither  instructor,  aircraft,  nor  pilot 
resources  were  sufficient  to  allow  for  meaningful  improvements 
over  normal  grading  practices  for  these  flights,  (fortunately, 
the  measurement  problems  was  less  Severe  for  the  T-39 
airborne  training  phase,  where  the  student  to  instructor 
ratio  was  one  to  one,  and  where  training  tasks  could  he 
better  defined  for  measurement  purposes.) 


The  preceeding  constraints  necessitated  placing  a 
heavier  emphasis  on  the  use  of  measures  of 'dead 'reckoning, 
which  could  be  obtained  within  the  context  of  training 


uuc  auch  measure  was  flrght  grades. 


— — - - 


as  it  existed. 

of  flight  grades  for  comparisons  of  student  performance  before 
versus  after  introduction  of  the  trainer  was  limited,  however, 
because  of  changes  in  the  application  of  grading  standards 
shortly  after  the  introduction  of  the  device.  Instructors 
felt  that  the  trainer  was  helping  to  produce  a  better 
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quality  product,  and  therefore  applied  grading  standards 
more  stringently  for  students  who  had  received  training 
in  the  device.  Because  of  this  "rubber  yardstick",  using 
grades  as  an  index  of  student  performance  could  be 
approached  in  only  a  very  limited  fashion. 

Because  of  the  above  measurement  constraints,  the  use 
of  other  types  of  evaluation  data  was  explored.  These 
included:  frequency  of  student  "downs"  (one  unsatisfactory 
grade  on  any  performance  element  on  any  flight) ;  frequency 
of  Training  Advisory  Board  (TAB)  reviews  of  students 
(required  following  two  "downs"  in  any  training  phase) ; 
and  attrition  rates  during  various  training  phase  (e.g. 
percent  of  students  not  completing  a  phase).  These  data 
were  not  utilized  for  several  reasons.  First,  they  were 
not  accessible  in  sufficiently  detailed  form  within  the 
temporal  and  fiscal  limitations  of  the  study.  It  was 
found,  for  example,  that  some  of  the  necessary  data  were  not 
centralized.  Where  data  were  centralized,  they  were  coded 
and  stored  in  keeping  with  quite  different  requirements , 
making  them  too  ambiguous  or  incomplete  for  meaningful 
research  application  in  a  study  such  as  the  one  reported 
here.  Finally,  the  utility  of  any  such  data,  including 
grades,  was  diminished  by  the  fact  that  the  calibre  of 
students,  as  measured  by  Aviation  Qualification  Tests  and 
Flight  Aptitude  Ratings,  was  declining  during  the  same 
period  when  Device  1D23  was  introduced. 

Ocher  measures  of  device  effectiveness  could  not  be  used 
because  of  the  philosophy  for  device  utilization.  Device  1D23 
was  not  intended  to  replace  or  be  used  as  a  substitute  for 
airborne  training.  Accordingly,  measures  of  device 
effectiveness  based  upon  savings  in  flight  training  were  not 
appropriate. 

Effectiveness  of  the  trainer  was  addressed  in  a  training 

context,  not  a  fleet  operational  context.  In-f light 
training  in  dead  reckoning  navigation  (T-29  and  il4 
aircraft)  and,  communications,  point-to-point  navigation 
and  airways  navigation  (T-39  aircraft)  were  selected  in 
conjunction  with  VT-10  personnel  as  the  mast  appropriate 
settings  in  which  to  measure  any  transfer  of  training  from 
the  device  to  related  t2.sks  in  airborne  settings.  The 
measurement  of  training  transfer  to  communication  and  nav¬ 
igation  tasks  in  jet  fighter  aircraft  (P-9  in-flicht 
training  phase)  was  not  addressed  because  only  two  flights 
were  involved  and  because  they  consisted  primarily  of  indoc¬ 
trination  to  low  level  visual  navigation,  formation  flight, 
and  aerobatics. 

As  a  result  of  the  above  constraints,  the  study 
utilized  flight  grades,  instructor  opinions,  and  student 
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opinions  for  assessment  of  the  effectiveness  of  the 
device  for  training  in  dead  reckoning  navigation.  For 
communications,  point-to-point,  and  airways  navigation, 
the  above  measures  were  supplemented  with  more  objective, 

[  instructor-generated  measures  of  student  perf ormance . 

Finally,  results  of  the  study  apply  to  the  utilization 
of  the  trainer  at  the  time  of  the  study  and  to  in-flight 
training  at  that  time.  What  is  trained  in  the  device, 
for  example,  is  highly  dependent  upon  the  content  of  pre¬ 
programmed  training  problems.  If  the  content  of  the 
training  problems  is  markedly  altered,  then  the 
effectiveness  of  the  device  may  change  accordingly. 
Similarly,  as  airborne  communication  and  navigation  task 
requirements  change,  the  effectiveness  of  the  trainer 
in  preparing  students  to  execute  these  tasks  may  change. 

The  balance  of  this  report  separately  addresses  each 
of  the  four  objectives  specified  at  the  beginning  of  this 
section.  Constraints  of  the  study  are  not  recited 
in  the  following  sections.  However,  interpretations  of 
i*  the  results  of  experimental  aspects  of  the  study  are 

L  presented  in  keeping  with  the  constraints. 


NAVTRAEQUIPCEN  72-C-0209-2 
SECTION  III 

TRAINING  EFFECTIVENESS  EVALUATION , 
DEAD  RECKONING  NAVIGATION 


OBJECTIVE 

The  objective  for  this  aspect  of  the  study  w-.?s  to  .\ssess 
impacts  of  training  in  Device  1D23  upon  student  dead 
reckoning  navigation  performance  as  demonstrated  during 
in-flight  training  in  the  T-29  or  C-114  aircraft. 

CONTEXT 

The  instructional  syllabus  established  that  the 
primary  purpose  of  T-29/C-114  training  flights  was  to 
introduce  student  NFOs  to  the  flight  environment  and  to 
make  relevant  all  navigational  instruction  (Ref  3}. 

Each  dead  reckoning  navigation  training  flight 
generally  was  comprised  of  seven  to  eight  students,  one 
instructor  NFO,  two  pilots  and  a  crew  chief.  Up  to  four 
students  performed  lead  navigator  dead  reckoning  tasks  for 
approximately  1.5  hours  during  ecch  training  flight,  which 
was  approximately  six  hours  in  duration.  When  net 
acting  as  lead  navigators,  studencs  performed  tracker 
dead  reckoning  navigation  tasks. 

METHOD 

Two  classes  were  selected  for  the  purpose  of 
comparing  dead  reckoning  flight  grades  before  versus  after 
the  introduction  of  training  in  the  device.  Class  408 
was  selected  because  it  provided  the  best  opportunity  to  examine 
flight  grades  before  instructors  had  much  collective  opportunity 
to  become  more  stringent  in  applying  grading  standards. 

Class  407  was  the  other  class  and  was  selected  for 
three  reasons.  First,  class  407  did  not  receive  training  in 
the  device.  Second,  it  received  training  in  the  immediate 
timeframe  of  class  408.  Third,  instructor  NFOs  agreed  that 
less  tangible  factors  such  as  class  spirit,  motivation, 
student  quality,  and  student  relationships  with  class  leaders 
and  advisors  were  comparable  for  classes  407  and  408. 

t 

Twenty- three  students  were  selected  from  each  class  for 
gT*''.de  comparison  purposes.  The  two  groups  were  quite 
similar  in  terms  of  distributions  of  Flight  Aptitude  Rating 
(FAR)  and  Aviation  Qualification  Test  (AQT)  scores.  FAR 
3tanine  scores  for  the  class  407  group  ranged  from  3  through 
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9,  with  a  mean  of  6.4.  AQT  stanine  scores  for  the  class  407 
group  ranged  from  4  through  9,  with  a  mean  of  7.2.  FAR  stanine 
scores  for  the  class  408  group  also  ranged  from  3  through  9, 
with  a  mean  of  6.2.  AQT  stanine  scores  for  this  group  ranged 
from  5  through  9,  with  a  mean  of  6.8. 

Students  in  each  class  received  five  training  flights, 
three  of  which  were  graded.  Using  standard  evaluation 
procedures,  instructor  NFOs  graded  each  student's  performance 
following  the  flights.  All  flights  were  conducted  in  keeping 
with  the  T-29  Flight  Syllabus  (Ref  3). 

Grading  was  done  after  each  flight  using  standardized 
Aviation  Training  Forms  and  grading  criteria  (Refs  3  and  4) . 
Using  a  four-point  scale,  a  numerical  grade  of  unsatisfactory 
(1) ,  below  average  (2) ,  average  (3) ,  or  above  average  (4) , 
was  determined  for  each  performance  element  which  was 
itemized  in  the  Aviation  Training  Form  and  which  was 
applicable  to  the  student's  requirements  during  the  flight. 
Performance  elements  for  the  training  flights  were:  positions, 
winds,  computations,  use  of  navigation  tools,  lead  navigator 
performance,  preflight,  logs  and  charts,  emergency  drill, 
response  to  instruction,  and  overall  performance.  As  is 
the  case  with  practically  all  such  grading  systems,  the 
resulting  grades  are  rather  subjective  because  of  latitudes 
which  exist  for  instructor  judgement  and  interpretation 
of  student  performance  in  relation  to  rather  generally 
defined  performance  elements. 

Overall  flight  grade  was  computed  for  each  student  in 
each  group  for  each  graded  flight  by  averaging  numerical 
grades  across  all  performance  elements.  Resulting  flight 
grades  for  the  two  groups  were  statistically  compared  for 
each  flight  as  well  as  across  all  flights.  The  factorial 
analysis  of  variance  statistical  test  v»as  used  (Ref  5) . 

As  a  separate-  activity,  instructor  NFOs  were  interviewed 
regarding  their  observations  and  opinions  of  student  per¬ 
formance  before  versus  after  the  initiation  of  training  in 
Device  1D23.  The  interviews  were  supplemented  by  a 
standardized  questionnaire. 

RESULTS 

FLIGHT  GRADES.  Table  1  presents  mean  flight  grades  for 
students  in  each  of  the  two  groups  for  the  three  graded 
flights  (flights  2,4,  and  5),  as  well  as  averages  across  flights. 
Mean  grades  for  the  class  4G8  group  (which  received  training 
in  the  device)  were  somewhat  higher  for  each  of  the  three 
flights,  as  was  their  average  across  flights.  Stat  j^afei^ally , 
however,  none  of  the  differences  between  groups  was 
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significant  at  the  .05  level1.  Althouqh  both  qroups  did  exhibit 
statistically  significant  (P  =  .01) 1  improvements 
in  grades  from  the  first  to  third  graded  flight. 

Rates  of  improvement  were  not  statistically  comparable  for 
the  two  groups.  It  can  be  concluded,  therefore,  that  both 
groups  performed  essentially  the  same  and  that  slightly 
better  grades  of  the  class  408  group  can  be  reasonably 
attributed  to  chance  factors  rather  than  training  in  Device  1D23. 


TAB  Li:  1.  AVERAGE  FLIGHT  GRADES,  DEAD  RECKONING  TASKS 


First 

Graded 

Flight 

Second 

Graded 

Flight 

Third 

Graded 

Flight 

Average 

Across 

Flights 

Class  407 

Group 

2.98 

3.05 

3.10 

3.04 

Class  408 

Group 

3.01 

3.13 

3.18 

3.10 

During  the  course  of  the  study,  several  instructor  NFOs  ^ 

indicated  that  they  felt  students  who  had  received  training 
in  the  device  were  achieving  more  above  average  grades  than 
students  who  had  not  received  such  training.  This  possibility 
was  explored  by  determining  the  number  of  above  average  grades 
received  by  each  student  in  either  group  on  any  performance 
element  during  the  three  graded  flights.  The  resulting 

information  is  summarized  in  Table  2.  i 

The  top  portion  of  Table  2  presents  mean  numbers  of  above 
average  grades  for  students  in  each  group.  Means  are 
separately  presented  for  each  graded  flight,  as  well  as 
averaged  across  all  flights.  On  the  first  and  third  flights, 
as  well  as  on  the  overall  flight  average,  students  in  the 
class  408  group  tended  to  receive  more  above  average  grades. 

Differences  between  group  means  were  statistically  compared 
using  the  t-test  for  independent  measures.  Comparisons  ' 

were  made  separately  for  eacc  xxignt  anu  ror  me  overall 
flight  averages.  Results  of  the  tests  were  far  from  being 


1  Indicates  the  probability  that  differences  between  the  two 
groups  could  have  resulted  from  chance  factors.  The  .05 
level  is  used  throughout  as  the  maximum  cutoff  probability 
for  determining  statistical  significance. 
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statistically  significant  at  the  .05  level.  It  can  be 
reasonably  concluded,  therefore,  that  the  observed  differences 
were  due  to  chance  factors. 

Mean  percents  of  grades  which  were  above  average  were 
similarly  compared  for  the  two  groups.  Again,  slight 
differences  appearing  to  favcr  the  class  408  group  *?ere 
not  statistically  significant. 

Percents  of  students  in  each  group  who  received  above 
average  grades  also  were  compared  separately  for  each 
graded  flight,  as  well  as  averaged  across  all  flights.  The 
procedure  recommended  by  Blowers  and  Lindquist  (Kef  6) , 
was  used  for  statistically  comparing  the  percentages. 

Results  of  the  tests  were  far  fro."  >^eing  significant  at  the 
.05  level.  It  can  be  reasonably  concluded,  therefore,  that 
minor  variations  between  groups  were  dufe  to  chance  factors. 

Although  none  of  the  analyses  of  flight  grades  showed 
statistically  reliable  improvements  in  student  performance 
on  airborne  dead  reckoning  tasks  as  a  result  of  training  in 
the  device,  instructor  questionnaire  responses  indicated 
that  numerous  improvements  were  evident. 


TABLE  2.  SUMMARY  OF  ABOVE  AVERAGE  GRADE 


DATA,  DEAD 

RECKONING 

TASKS 

First 

Second 

Third 

Average 

Graded 

Graded 

Graded 

Across 

Flight 

Flight 

Flight 

Flights 

Mean  No.  of  407*  1.13  1.83  1.70  1.55 

Above  Average 

S  Grades  Per  408°  1.21  1.78  2.26  1.75 

Student 


Mean  Percent 

407 

11.4 

17.6 

16.6 

1.52 

of  Grades 

Which  Were 

408 

11.9 

17.6 

22.7 

1.74 

Above  Average 

Percent  of 

407 

78.2 

69.5 

78.2 

75.3 

Students  with 
Above  Average 
Grades 

408 

65.2 

78.2 

82.6 

75.3 

*  Class  groups 

INSTRUCTOR  QUESTIONNAIRE  RESPONSES.  '  A  frequently  overlooked 
source  of  valuable  information  is  individuals  who  have 
extensive,  day-to-day  experience  with  the  device  of  procedure 
being  evaluated.  In  the  present  study,  steps  were  taken  to 
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ensure  that  instructor  observations,  opinions  and  viewpoints 
would  net  be  overlooked.  This  was  done  by  interviewing  a 
number  of  instructor  NFOs  and  consolidating  content  of  the 
interviews  and  other  information  items  -co  form  a  standard 
instrument  (questionnaire) .  Rating  scales  used  in  the 
questionnaire  were  reviewed  with  a  sample  of  instructors 
to  ensure  that  they  had  meaning  to  the  instructional 
community. 

The  resulting  questionnaire,  shown  in  Appendix  B, 
was  completed  by  all  instructor  NFOs  and  instructor  pilots 
who  w**”e  involved  with  training  in  Device  1D23,  T-29,  C-jl14, 
or  x— 35  fti2~r?,ft  during  the  period  when  the  device  was 
introduced. 

The  questionnaires  were  cdxapletsd  at  the  end  of  the 
data  collection  period.  Fourteen  instructor  NFOs  responder'  to 
elements  of  the  questionnaire  which  were  applicable  to 
transfer  of  training  from  Device  1D23  to  airborne  dead 
reckoning  navigation  tasks.  Their  responses  are  summarized 
below. 


Instructor  NFOs  were  asked  to  rate  how  much  value  they 
relfc  training  in  Device  1D23  had  in  preparing  students  for 
T-29/C-114  training  flights.  Response  options  were: 
negative  value,  no  value,  minimal  value,  moderate  value, 
and  high  value.  Fourteen  percent  responded  that  the 
training  was  of  moderate  value.  The  remaining  86  percent  indi 
cated  that  the  training  had  high  value. 


The  second  question,  also  of  an  overall  nature,-  was 
intended  to  examine  the  general  magnitude  of  any 
improvements  in  student  performance  which  may  have  been 
evident  following  the  introduction  of  training  in  Device 
1D23.  To  do  this,  instructors  were  asked  to  compare  general 
student  performance  on  each  of  the  training  flights  before 
and  after  introduction  of  the  device.  A  majority  (71  percent) 
indicated  that  overall  student  performance  on  the  first  dead 
reckoning  training  flight  following  introduction  of  the  device 
was  equivalent  with  performance  which  used  to  be  observed  on 
the  third  flight  before  the  device  was  introduced.  A  majority 
(73  percent)  also  rated  the  second  flight  following  training  an 
the  device  as  equivalent  with  the  third  flight  before  the  device. 
Ninety  percent  rated  the  third  flight  following  training 
in  the  device  as  equivalent  with  the  fourth  flight  prior 
to  the  device.  Finally,,  67  percent  rated  the  fourth  flight 
following  training  in  the  device  as  being  equivalent  with 
the  fifth  flight  prior  to  the  device. 
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Irrespective  of  iack  of  differences  associated  with 
grades,  instructor  NFOs  were  consistent  in  their  views  that 
overall  improvements  in  student  dead  reckoning  performance 
resulted  from  training  in  the  device.  A  thirj 
questionnaire  item  was  designed  to  escaraine  these  views 
in  greater  detail.  The  item  conjisted  rt  38  task  activities 
required  of  students  during  dead  reckoning  ravigation  training. 
Tht.  activities  were  clustered  under  four  main  topic  areas. 
Instructors  were  asked  to  rate  hew  well  they  felt  training 
in  the  device  prepared  students  to  execute  each  of  the  task 
activities. 

Table  3  summarizes  results  of  the  ratings.  An  "X” 
in  the  columns  of  the  table  shows  median  instructor  ratings. 

It  is  notable  that  none  of  the  consensus  (median)  ratings 
fell  within  the  categories  of  hurts  greatly  or  hurts  some. 

Only  three  task  activities- were- rated  as  having  not' been 
influenced-  by  training  in- the  devices  (Ho\! Effect  category). 

These  were:  identifying  restricted  areas,  preflighting 
logs,  and  preparing  flight  plans.  Median  ratings  for  all 
remaining  task  activities  indicated  that  they  benefited  some 
or  greatly  as  a  result  of  training  in  the  device. 

Instructors  also  were  asked  to  address  the  subject  of 
substituting  additional  trainer  time  for  flight  time. 

To  best  interpret  instructor  responses,  it  must  be  pointed 
out  that  the  fifth  dead  reckoning  training  flight  was 
eliminated  shortly  after  training  in  the  device  was  instituted. 
Elimination  of  the  flight  should  not  be  attributed  to  the 
trainer,  however.  It  had  been  felt  for  some  time  that  the 
fifth  flight  was  not  productive,  and  instructors  indicated 
that  the  flight  would  have  been  eliminated  regardless  of 
the  trainer.  None  the  less,  some  reduction  of  in-flight 
training  had  occurred  in  the  general  timeframe  during 
which  instructors  were  asked  to  address  the  possibility  of 
additional  cuts  in  in-flight  training. 

Instructor  responses  were  about  evenly  split  when  asked 
whether  they  telt  additional  training  in  Device  1D23  could 
be  substituted  for  any  of  the  dead  reckoning  training  flights. 
Fifty-four  percent  responsed  yes  ;  46  percent  responded  no. 

The  instructors  were  asked  to  elaborate  upon  their 
responses  to  substituting  trainer  time  for  flight  time. 

Those  who  felt  that  flight  time  could  not  be  compenlated 
for  by  additional  trainer  time  cited  numerous  factors 
associated  with  what  cypically  is  called  "the  realism  of 
flight."  The  factors  generally  fell  into  two  related 
categories.  One  category  centered  upon  th»  psychological 
pressures  of  flight,  including:  correctly  communicating 
with  the  pilot  and  crew;  being  in  charge  of  an  aircraft's 
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flight  path?  stress  resulting  from  noise  and  temperature;  and 
having  to  continue  performing  in  spite  of  fatigue,  nausea, 
uncertainty,  hazards,  or  emergencies.  A  second  category 
centered  upon  more  physical  aspects  of  flight,  including: 
aircraft  movement  and  responses  to  turbulence;  in-flignt 
emergencies;  equipment  malfunctions  and  failures;  marginal 
reception  on  navigation  aids,  including  the  breaking  of 
lock  on  stations;  noisy  communication  channels?  receiving 
erroneous  instrument  readings  from  pilots*  and  pilot 
errors  in  flying  command  headings.  Content  of  the  second 
category  appears  to  offer  a  number  of  suggestions  for  ways 
to  modify  training  in  Device  1D23  to  enhance  its  utility 
in  preparing  students  for  inflight  performance. 

TABLE  3.  SUMMARY  OF  TASK  ACTIVITY  RATINGS,  DEAD  RECKONING 


—  — — - - - - - - 

PREPLIGHT  PLANNING  &  PREPARATION 

Interpreting  charts . 

Identifying  restricted  areas  . 

Hurts  Greatly 
Hurts  Some 

No  Effect 

Helps  Some 
Helps  Greatly 

V 

Plotting  courses  .  .  . 

X 

Using  correct  symbology . 

X 

Selecting  radio  nav  aids  . 

X 

[I 

Preflighting  logs . 

Preparing  flight  plans  . 

X 

□ 

X 

MEASURING  AND  COMPUTING 

_1 

Using  CR  2/3  computer . 

X 

Using  plotter  &  divider  . 

X 

Determining  TAS  . 

IT 

Determining  ground  speed  .  .  .  .  . 

X 

Determining  ETA  . 

X 

Determining  wind  direction/velocity.  „  „  .  . 
Determining  drift  angle,  ,  .  . 

~ 

NAVIGATING 

Dead  reckoning  procedures . 

lx 

Understanding  radials  from  nav  aids . 

Plotting  accurate  TACAN  fixes,  ....... 

Plotting  multiple  LOPs  ,  . 

r 

r 

Advancing/Retardin^  LOPs  .  ,  . 

”ir 

Plotting  EPs . . 

"" 

L_£_ 

Applying  variation  . 

X  j 

Converting  between  mag  &  true  heading.  .  .  . 
Plotting  track,  no-wind,  wind  lines,  .  ,  .  . 
Using  correct  DR  symbologv . .  ,  . 

X 

X 
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TABLE  3.  SUMMARY  OF  TASK  ACTIVITY  RATINGS,  DEAD  RECKONING  (Cont) 


NAVIGATING  (continued) 

Dead  reckoning  ahead  .  ,  .  .  . 
Correctly  applying  draft  angle 
Determining  headings  to  fly  . 

Filling  out  logs  . 

Communicating  with  pilot  .  .  . 


INTEGRATION  OF  KNOWLEDGE 

Getting  it  all  together  . 

Interpreting  instruction  ........ 

Keeping  oriented  relative  to  aircraft 
position 

Keeping  oriented  relative  to  direction  . 
to  checkpoints 

Pacing  tasks  to  "keep  ahead  of  the  .  .  . 
aircraft" 

Anticipating  what  would  occur  in  flight 

Understanding  spatial  relationships  .  . 

Identifying  incorrect  Nav  inputs  or  .  „ 
solutions 

Identifying  procedural  and  computational 
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X 
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Instructors  who  indicated  that  trainer  time  might  be 
substituted  for  flight  time  were  asked  to  comment  on  what 
additional  training  in  the  device  would  be  appropriate. 

A  majority  of  those  responding  suggested  additional 
training  which  would  be  patterned  after  the  lead-tracker 
navigator  structure  used  in  dead  reckoning  training  flights. 
They  pointed  out  that  such  additional  training,  however, 
could  be  valuable  only  if  each  student  was  afforded  a 
reasonable  opportunity  to  act  as  lead  navigator  and  if 
sufficient  instructional  personnel  were  available  to  work 
with  and  debrief  students.  A  second  recommendation  was  to  cut 
in-flight  training  in  half,  and  provide  extra  trainer  or 
in-flight  practice  only  for  weak  students.  Implementation  of 
this  recommendation,  of  coarse,  would  require  improved  methods 
for  assessing  student  proficiency.  Along  this  line,  a 
recommendation  also  was  made  to  substitute  one  graded 
flight  for  a  graded  trainer  session.  This  recommendation  was 
predicated  upon  using  the  trainer's  computer  and  software 
for  objective,  unbiased  evaluation  of  student  proficiency. 
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Instructors  who  indicated  that  trainer  time  could  not 
be  substituted  for  flight  time  were  asked  to  suggest  modifica¬ 
tions  to  the  trainer  and  to  TPs  which  miqht  make  it  possible 
to  make  such  substitutions.  A  number  of  recommendations 
were  made,  including:  improve  student-to-instructor  ratio 
to  facilitate  the  diagnosis  of  student  learning  problems  and 
allow  for  more  adequate  debriefings;  add  static  and  communica- 
tions  patter  to  radio  communications  simulation;  degrade 
simulated  reception  on  radio  navigation  aids,  to  include 
breaking  of  lock  at  appropriate  aircraft  headings  and 
attitudes;  incorporate  unexpected  route  changes,  such  as 
those  resulting  from  vectoring;  ensure  that  students  all 
have  sufficient  opportunities  to  act  as  lead  navigators 
in  training  patterned  after  the  lead-tracker  navigator 
structure  used  in-f light  •  and  add  a  motion  capability  to  the 
trainer. 

In  two  separate  questions,  instructors  were  asked 
to  separately  list  elements  of  training  in  Device  1D23 
which  they  felt  had  the  best  value  or  the  least  value  in 
preparing  students  for  in-flight  dead  reckoning  trair.irg. 
Factors  frequently  listed  as  best  values  included: 
training  in  procedures  and  motor  skills  associated 
with  the  use  of  navigation  tools;  forcing  students  to 
operate  under  tine  pressures,  pace  their  tasks, 
time-share  among  tasks,  and  organize  tasks  on  priority 
bases;  training  in  the  use  of  flight  logs,  including 
which  items  are  most  important  and  how  to  obtain  them; 
and  preparing  students  for  the  lead-tracker  roles  which 
they  assume  in  flight  training.  Less  frequently  listed 
values  were:  training  in  the  selection  of  suitable 
navigation  aids;  development  of  an  improved  understanding  of 
radials  and  spatial  relationships;  and  facilitating  student 
interpretation  of  instruction. 

Factors  listed  as  having  the  least  value  included: 
voice  communications  training;  training  as  lead 
navigators  in  training  problems  designed  to  emulate  the 
lead-tracker  navigator  structure  used  in  dead  reckoning 
training  flights;  and  student  log  keeping  and  computational 
practice.  Both  of  the  latter  factors  were  affected  by  the 
inability  of  instructors  to  devote  sufficient  time  to 
monitoring  individual  student  performance  because  of 
characteristically  high  student-to-instructor  ratios. 

Finally,  instructors  were  asked  to  rate  the  concept 
of  using  Device  1D23  to  train  all  students  to  at  least  the 
same,  standardized  performance  levels  prior  to  allowing 
them  to  advance  to  in-flight  dead  reckoning  training. 

Response  options  were:  very  undesirable,  possibly  undesirable, 
no  opinion,  possibly  desirable,  and  very  desirable.  Fourteen 
percent  rated  the  concept  as  possibly  undesirable;  22  percent 
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rated  as  possibly  desirable;  and  64  percent  as  very  desirable. 
Several  apprehensions  were  expressed,  however,  even  by 
individuals  who  rated  the  concept  as  desirable.  The 
apprehensions  centered  around  the  ability  to  objectively 
determine  student  proficiency  in  the  device  and  the 
possibility  that  students  would  queue  up  due  to  inabilities 
to  demonstrate  proficiency,  thus  complicating  student 
scheduling.  One  instructor  also  pointed  out  that  a 
small  percentage  of  students  do  not  appear  to  fully  grasp 
the  meaning  of  navigation  tasks  until  they  are  exposed 
to  in-flight  training;  therefore,  placing  too  strong  an 
emphasis  on  evaluating  them  in  the  device  could  be  unfair. 

CONCLUSIONS 

Based  solely  upon  statistical  analysis  of  flight  grades, 
it  would  have  to  be  concluded  that  training  in  Device  1D23 
had  no  effect  upon  the  proficiency  with  which  students  were 
able  to  execute  dead  reckoning  navigation  tasks  in  an  airborne 
setting.  It  is  felt,  however,  that  reliable  conclusions 
regarding  the  training  effectiveness  of  Device  1D23  cannot 
be  drawn  from  flight  grades  alone. 

Regardless  of  attempts  to  achieve  objectivity  and 
standardization,  grades  are  basically  subjective  measures 
of  performance.  As  such,  they  may  not  possess  necessary 
research  sensitivity  to  detect  the  effects  of  changes  in 
instruction.  Flight  grades  were  used  in  the  present  study 
primarily  because  they  constituted  the  only  practically 
available  measure  of  in-flight  dead  reckoning  performance. 

They  also  were  used  because  the  possibility  existed  that 
standardized  student  evaluation  practices  might  have 
rendered  them  sufficiently  sensitive  to  be  of  research 
value.  Based  upon  other  information  developed  during  this 
aspect  of  the  study,  it  is  felt  that  grades  were  not 
sufficiently  sensitive  to  warrant  the  conclusion  that 
training  in  the  device  did  not  result  in  improved  dead 
reckoning  performance. 

The  strength  and  consistency  of  instructor  opinion 
data  suDport  the  conclusion  that  training  in  Device  1D23 
has  rebuiled  in  improvements  in  virtually  every  aspect  of 
student  confidence  and  performance  in  the  execution  of 
dead  reckoning  tasks  in  an  airborne  setting. 

Evidence  also  was  found  to  support  the  conclusion 
that  the  first  six  training  sessions  in  the  device 
(including  two  purely  introductory  sessions)  were 
approximately  the  equivalent  of  the  first  two  dead 
reckoning  training  flights.  This  conclusion  is  based 
upon  the  consensus  instructor  viewpoint  that  overall 
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student  performance  on  the  first  flight  after  the 
introduction  of  training  in  the  device  was  equivalent  with 
overall  strdent  performance  which  used  to  be  observed  on 
the  third  flight  prior  to  the  trainer. 

Based  upon  the  above  information,  it  would  also 
appear  reasonable,  on  an  exploratory  basis,  to  reduce  the 
number  of  dead  reckoning  training  flights  from  five 
to  anDroximately  three. 

It  also  appears  reasonable  to  conclude  that  the 
training  effectiveness  of  the  device  might  be  improved 
by  effecting  changes  in  device  software  to  better 
simulate  characteristics  of  the  in-flight  environment 
(e.g.,  noisier  voice  communications  channels  and  marginal 
radio  navigation  aid  signals),  improve  the  device's 
automated  performance  measurement  capability,  and  provide 
additional  lead  navigator  practice  during  simulations 
of  T- 2 9/C- 114  training  flights. 
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SECTION  IV 

TRAINING  EFFECTIVENESS  EVALUATION, 
T-39  PHASE 


OBJECTIVE 

The  objective  for  this  aspect  of  the  study  was  to 
determine  impacts  of  training  in  Device  1D23  upon  student 
communication  and  navigation  task  performance  in  the  T-39 
airborne  training  phase  (point-to-point  and  airways  navigation) . 

CONTEXT 

The  instructional  syllabus  (Ref  7)  established  that  a 
primary  purpose  of  the  T-39  training  flights  was  to  enable 
students  to  apply  all  basic  navigation  procedures  for 
Tacan  point-to-point  and  airways  navigation  while  in  a  jet 
flight  environment.  Additionally,  the  flights  were  designed 
to  provide  students  with  cockpit  familiarization  and 
additional  instruction  in  mental  dead  reckoning  and 
communications  procedures. 

Each  T-39  training  flight  consisted  of  three  students, 
one  instructor  NFO  and  an  instructor  pilot.  Three  training 
flights  were  required  for  each  student.  Each  flight, 
summarized  below,  varied  in  terms  of  student  task  requirements. 

The  first  T-39  training  flight  was  a  point-to-point 
flight  consisting  of  three  segments,  each  of  which  required 
approximately  45  minutes.  During  each  segment,  students 
would  perform  one  of  three  task  clusters.  One  cluster 
involved  navigation  of  the  aircraft  and  execution  of  fuel 
management  procedures.  The  second  cluster  involved 
communications  outside  the  aircraft,  and  included  cockpit 
familiarization.  The  third  cluster  involved  tracker 
navigation  and  maintaining  a  jet  log.  Students  rotated 
positions  in  the  aircraft  so  that  each  student  was  provided 
an  equal  opportunity  to  perform  the  various  task  clusters 
during  one  of  the  three  flight  segments. 

The  second  flight  was  an  airways  navigation  flight. 

It,  too,  was  divided  into  three  45  minute  segments  to  allow 
each  student  an  equal  opportunity  to  perform  differing  task 
clusters.  One  cluster  required  navigation  of  the  aircraft, 
execution  of  fuel  management  procedures,  and  communication 
outside  the  aircraft.  The  second  cluster  involved  executing 
copilot  duties.  The  third  cluster  involved  tracker 
navigation  and  maintaining  a  jet  log.  Students  rotated 
positions  in  the  aircraft  so  that  each  was  provided  an  equal 
opportunity  to  perform  the  various  task  clusters  during  one 
of  the  three  flight  segments. 
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The  third  flight  actually  consisted  of  three  mini 
airways  flights,  each  of  which  was  approximately  one  hour 
in  duration.  During  each  mini  flight,  one  student  performed 
all  task  clusters,  from  takeoff  through  landing,  from  the 
copilot's  seat.  The  remaining  two  students  performed 
tracker  navigation  tasks. 

During  all  flights,  when  a  student  was  acting  as  the 
lead  navigator  or  when  he  was  performing  communications  or 
copilot  duties,  he  was  under  the  direct  observation  of  an 
instructor  NFO  or  instructor  pilot.  This  one-to-one  student- 
to-instructor  ratio  greatly  facilitated  the  measurement  of 
student  performance. 

METHOD 

Two  classes  1407  and  408)  were  selected  for  the  purpose 
of  comparing  flight  grades  before  versus  after  the  introduction 
of  training  in  Device  1D23.  They  were  selected  for  reasons 
previously  discussed  in  Section  III.  Twenty-three  students 
were  selected  from  each  class  for  grade  comparisons.  They 
were  the  aame  students  selected  for  comparison  of  dead 
reckoning  flight  grades. 

In  anticipation  of  the  training  effectiveness  evaluation 
and  the  need  for  more  objective,  task-oriented  measures  of 
student  performance,  Navy  personnel,  including  instructors 
at  VT-10,  developed  a  supplemental  performance  evaluation 
form.  Content  of  the  form  is  defined  in  Table  4.  Student 
errors  were  recorded  on  supplemental  data  forms  as  they 
occurred  in  flight. 

The  forms  were  used  in  addition  to  flight  grades  to 
supplement  the  evaluation  of  performance  of  students  in  class 
407  during  their  T-39  training  flights.  Since  this  class 
did  not  receive  training  in  the  device,  performance  data 
developed  from  the  forms  provided  a  baseline  against  which  to 
compare  performance  of  a  class  which  had  received  training 
in  the  device. 

Class  416  was  selected  for  comparison  with  class  407 
using  the  supplemental  measures  of  student  performance. 

Class  416  was  selected  for  four  reasons.  First  class  416 
had  received  training  in  the  device.  Second,  the  class  was 
accessible  for  performance  measurement  within  the  timeframe 
of  the  study.  Third,  instructors  agreed  that  less  tangible 
factors  such  as  class  spirit,  motivation,  student  quality, 
and  student  relationships  with  class  leaders  and  advisors 
were  comparable  with  those  of  class  407.  Fourth,  it  was 
possible  to  select  a  group  of  23  students  from  class  416 
which  was  comparable  with  the  class  407  group  in  terras  of 
FAR  and  AQT  scores.  For  the  class  407  group,  F£R  stanine 
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TABLE  4.  CONTENT  OF  SUPPLEMENTAL  PERFORMANCE 
EVALUATION  FORMS 


Measure 


Description 


Number  of  Voice 
Communication  Errors 

Number  of  times  when  a  student 
made  content  or  format  errors 
during  a  radio  transmission. 

Number  of  Missed 

Calls 

Number  of  times  when  a  student 
tasked  with  monitoring  voice 
communications  failed  to 
identify  a  transmission  to 
the  aircraft. 

ETAs,  Number  of 

Minutes  Off 

Average  error  in  minutes 
between  coraouted  estimated 
time  of  arrival  and  actual 
time  of  arrival . 

Number  of  Wrong 

Way  Turns 

Number  of  times  when  a  student 
directed  the  pilot  to  turn  the 
aircraft  in  a  direction 
opposite  that  which  was  required 

Number  of  Heading 

Errors 

Number  of  times  when  a  student 
directed  the  pilot  to  fly  a 
heading  which  was  at  least 

10  degrees  different  from  the 
heading  which  was  required. 

Number  of  No.  2 

Needle  Reading  Errors 

Number  of  times  when  a  student 
misread  the  displayed  value 
of  the  No.  2  needle  of  the  Radio 
Magnetic  Indicator  by  at  least 

5  degrees. 

Number  of  Altimeter 
Reading  Errors 

Number  of  times  when  a  student 
misread  the  altimeter  by  at 
least  1,000  feet. 

Number  of  Departure 
Errors 

Number  of  times  when  a  student 
failed  to  or  was  unable  to 
direct  the  pilot  through  normal 
instrument  departure  and 
clearance  compliance  procedures. 
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TABLE  4.  CONTENT  OF  SUPPLEMENTAL  PERFORMANCE  EVALUATION  FORMS 

(continued) 


Measure 


Description 


Number  of  Enroute 
Errors 


Number  of  Turn 
Point  Errors 


Number  of  Fuel 
Management  Errors 


Number  of  Approach 
Errors 


Minutes  to  get  on 
Radial 


Number  of  times  when  a  student 
failed  to  or  was  unable  to 
direct  the  pilot  through 
radial  tracking,  radial  intercept, 
station  passage  or  wind  correction. 

Number  of  times  when  a  student 
failed  to  or  was  unable  to 
execute  standard  turn  point 
procedures  of  giving  the 
pilot  the  new  course,  obtaining 
fuel  flow  and  quantity  informa¬ 
tion,  giving  the  pilot  a  new 
heading  to  fly  at  minimum  DME, 
directing  the  aircraft  onto 
the  outbound  radial,  and  computing 
estimated  ti-je  of  arrival  at 
the  next  turn  point. 

Number  of  times  when  a  student 
failed  to  or  was  unable  to 
correctly  determine  fuel 
required  for  the  next  leg  and 
fuel  required  for  the  balance 
of  the  flight. 

Number  of  times  when  a  student 
failed  to  or  was  unable  to 
direct  the  pilot  through  normal 
instrument  approach  and  clearance 
compliance  procedures. 

Number  of  minutes  between  the 
time  when  a  student  should  have 
begun  to  intercept  a  radial 
(eg. ,  at  a  turn  point)  until 
he  had  directed  the  aircraft 
onto  the  radial,  averaged  across 
all  radial  intercepts  by  a 
student  during  a  flight. 
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TABLE  4.  CONTENT  OF  SUPPLEMENTAL  PERFORMANCE  EVALUATION  FORMS 
_ _ (continued) _ 

Measure  Description 


Percent  of  Time  on  Radial  Number  of  minutes  a  student 

maintained  the  aircraft  on  a 
radial  divided  by  the  total 
number  of  minutes  required  to 
fly  the  leg,  averaged  over  all 
radials  tracked  by  a  student 
during  a  flight. 


scores  ranged  from  3  through  9,  with  a  mean  of  6.4.  AQT 
stanine  scores  ranged  from  4  through  9,  with  a  mean  of  7.2. 

For  the  23  students  selected  from  class  416,  FAR  stanine 
scores  also  ranged  from  3  through  9,  with  a  mean  of  6.4.  AQT 
stanine  scores  ranged  from  4  through  9,  with  a  mean  of  7.0. 

Students  in  each  class  group  received  three  training 
flights,  all  of  which  were  graded.  Supplemental 
performance  measures  were  collected  for  all  students  in 
classes  407  and  416.  All  flights  were  conducted  in 
keeping  with  the  T-39  flight  syllabus  (Ref  7) . 

Grading  was  done  after  each  flight  using  standardized 
Aviation  Training  Forms  and  grading  criteria  (Refs  7  and  8) . 
The  four  point  scale  previously  described  in  Section  III 
was  used  to  determine  a  grade  for  each  performance  element 
which  was  itemized  in  the  Aviation  Training  Form  and  which 
was  apolicable  to  the  student's  requirements  during  the 
flight.  Performance  elements  for  the  training  flights  were: 
preflight  planning,  communications  procedures,  tAlAN  point- 
to-point,  fuel  management,  use  of  radio  navigation  aids, 
use  of  FLIPS  and  publications,  departure  procedures, 
enroute  procedures,  approach  procedures,  emergency  procedures, 
preflight  and  postflight  inspections,  instrument  interpretation 
mental  dead  reckoning,  response  to  instruction,  and  aircraft 
servicing. 

An  overall  flight  grade  was  computed  for  each  student 
in  the  class  407  and  408  groups  for  each  flight.  This  was 
done  by  averaging  numerical  grades  across  all  performance 
elements.  Resulting  flight  grades  for  the  two  groups  were 
statistically  compared  for  each  flight  as  well  as  across  all 
flights.  The  factorial  analysis  of  variance  statistical  test 
was  used  (Ref  5) . 
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Data  derived  from  the  supplemental  performance 
evaluation  forms  were  subjected  to  statistical  test  using 
both  univariate  (factorial  analysis  of  variance)  and 
multivariate  (multiple  discriminant  analysis)  techniques 
(Ref  5).  The  multiple  discriminant  analysis  technique 
is  briefly  described  in  Appendix  C. 

As  a  separate  activity,  instructor  NFOs  were  interviewed 
regarding  their  observations  and  opinions  of  student  perform¬ 
ance  during  T-39  flights  before  versus  after  the  initiation 
of  training  in  Device  1D23.  The  int^Mviews  were  supplemented 
by  a  Questionnaire . 

RESULTS 

FLIGHT  GRADES.  Table  5  presents  mean  flight  grades  for 
students  in  the  class  407  and  408  groups  for  each  of  the 
T-39  flights,  as  well  as  averaged  across  all  flights.  Mean 
flight  grades  were  identical  for  both  groups  on  the  first 
flight.  On  the  second  flight,  grades  were  slightly  higher 
for  the  class  407  group  (which  had  not  received  training 
in  the  device) .  Grades  were  virtually  identical  for  both 
groups  on  the  third  flight.  Averaged  across  all  flights, 
grades  were  slightly  higher  for  the  class  407  group. 

The  analysis  of  variance  statistical  test  of  the  grades 
revealed  that  none  of  the  differences  between  groups 
was  significant  at  the  .05  level2.  It  can  be  concluded, 
therefore,  that  slight  differences  in  flight  grades  between 
the  two  groups  can  be  reasonably  attributed  to  chance  factors 
rather  than  to  effects  of  training  in  Device  1D23. 

TABLE  5.  MEAN  FLIGHT  GRADES, 

T-39  FLIGHTS 


First 

Flight 

Second 

Flight 

Third 

Flight 

Average 

Across 

Flights 

Class 

Group 

407 

3.02 

3.13 

3.06 

3.07 

Class 

Group 

408 

3.02 

2.99 

3.04 

3.02 

2 Indicates  the  probability  that  differences  between  the  two 
groups  could  have  resulted  from  chance  factors.  The  .05 
level  is  used  throughout  as  the  maximum  cutoff  probability 
for  determining  statistical  significance. 
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SUPPLEMENTARY  PERFORMANCE  MEASURES.  Supplementary  performance 
data  previously  described  in  table  4  were  statistically 
analyzed.  For  the  analyses,  numbers  of  wrong  way  headings 
and  heading  errors  in  excess  of  10  degrees  were  combined  into 
a  single  measure  called  number  of  wrong  headings.  Number 
of  minutes  to  get  on  the  radial  and  percent  of  time  on  radial 
were  analyzed  only  for  the  second  and  third  flights.  The 
first  flight  was  not  an  airways  flight,  and  the  measures 
were  not  applicable. 

The  supplementary  data  were  analyzed  using  multiple 
discriminant  analysis  and  factorial  analysis  of  variance. 

One  set  of  analysis  of  variance  tests  was  used  to  compare 
student  performance  separately  for  each  of  the  three  flights 
as  well  as  across  flights.  This  set  of  tests  was  computed 
for  each  measure  except  time  to  get  on  radial  and  percent  of 
time  on  radial.  An  additional  set  of  analysis  of  variance 
tests  was  computed  to  compare  student  performance  separately 
for  the  second  and  third  flights  as  well  as  across  both 
flights  for  all  13  measures.  Results  obtained  from  the  multiple 
discriminant  analyses  and  the  analysis  of  variance  tests  were 
virtually  identical.  Accordingly,  only  results  of  the 
analysis  of  variance  tests  are  reported. 

Two  measures  showed  hiqhly  comparable  rates  of  change 
between  the  first  and  third  training  flights  for  the  two 
groups.  Altimeter  reading  errors  for  both  groups  increased 
from  an  average  of  ,37  per  student  on  the  first  flight  to  an 
average  of  .39  per  student  on  the  third  flight.  This  change 
was  statistically  significant  at  the  .05  level.  Turn  point 
procedural  errors  for  both  groups  decreased  from  an  average 
of  1.65  per  student  on  the  first  flight  to  1.00  per  student 
or.  the  third  flight.  This  change  also  was  statistically 
significant  at  the  .05  level. 

Statistical  analysis  also  showed  that  several  measures 
exhibited  differential  rates  of  change  across  flights  for 
the  two  groups.  Rates  of  change  of  number  of  RMI  number 
two  needle  reading  errors  were  similar  for  both  groups 
between  the  first  and  second  T-39  flights.  Between  the 
second  and  third  flights,  however,  average  numbers  of 
reading  errors  per  student  for  the  class  407  group  increased 
from  .22  to  .83.  The  average  number  of  reading  errors  per 
student  for  the  class  416  group  increased  only  from  .04  to  .09. 

Similarly,  numbers  of  fuel  management  errors 
were  statistically  comparable  for  both  groups  on  the  first 
flight.  However,  the  class  416  group  exhibited  significantly 
(P  »  .05)  better  performance  on  both  second  and  third 
flights. 
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ETA  errors  were  significantly  {P  =  .01)  smaller  for  the 
class  406  group  only  on  the  first  flight.  On  the  second  and 
third  flights,  both  groups’'  errors  were  statistically 
comparable. 

In  a  reverse  trend,  numbers  o ,*  turn  point  errors 
were  comparable  for  both  groups  on  the  first  flight.  The 
class  407  group  showed  no  meaningful  decline  in  these 
errors  through  the  third  flight,  while  class  416  group 
errors  declined  from  an  average  of  1.91  errors  per  student 
on  the  first  flight  to  .52  errors  per  student  on  the  third 
flight. 

Departure  errors  also  exhibited  statistically  (P  =  .01) 
different  trends  across  flights  for  the  two  groups.  The  Mean 
number  of  errors  per  student  in  the  class  407  group  was  .26 
on  the  first  flight,  rising  to  .78  on  the  second  flight,  and 
then  declining  to  .52  on  the  third  flight.  For  the  class  416 
group/ mean  errors  were  higher  on  the  first  flight  (.52  errors 
per  student),  but  they  declined  sharply  to  .04  on  the  second 
flight.  Errors  then  rose  to  am  average  of  .70  per  student 
on  the  third  flight.  Reasons  for  the  marked  divergence 
of  error  patterns  for  the  two  groups  is  not  clear. 

Averaged  across  flights,  the  difference  between  groups 
was  not  statistically  significant. 

None  of  the  remaining  six  measures  showed  any  statistically 
reliable  changes  across  flights. 

Averaged  across  flights,  statistically  significant 
performance  improvements  were  found  on  10  of  the  13  measures 
for  the  class  416  group.  Table  6  summarizes  data  of 
relevance  to  the  ten  measures.  For  each  measure,  mean  values 
per  student  in  each  group  are  presented  in  columns  one  and 
two.  Column  three  displays  percent  improvements  of  the 
class  416  group  over  the  class  407  group  for  each  measure. 

The  measures  are  ranked  in  terms  of  percent  improvement,  which 
ranged  from  a  high  of  84  percent  to  a  low  of  17  percent  for  the 
ten  measures.  The  overall  average  improvement  was  41  percent. 

The  fourth  column  presents  approximate  probabilities  that  the 
performance  improvements  could  have  resulted  from  chance  factors. 

No  significant  differences  in  performance  between  the 
two  groups  were  found  for  three  of  the  13  measures. 

Table  7  summarizes  data  of  relevance  to  the  three  measures. 

In  the  table,  percent  change  values  are  given,  because  not 
all  changes  reflected  improvements  as  a  result  of  training 
in  the  device. 
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Mean  number  of  departure  errors  was  slightly  less 
for  the  group  which  received  training  in  the  device.  Mean 
numbers  of  enroute  errors  and  approach  errors,  however,  were 
larger  for  the  trainer  (class  416)  group.  Although 
differences  between  groups  appear  relatively  large, 
particularly  in  terms  of  percent  change,  variability  inherent 
in  these  measures  apparently  was  sufficiently  large  to 
preclude  the  differences  from  being  statistically  significant. 
It  must  be  concluded,  therefore,  that  differences  between 
groups  in  terms  of  numbers  of  departure,  enroute  and 
approach  errors  were  due  to  chance  factors. 

TABLE  6.  SUMMARY  OF  PERFORMANCE  IMPROVEMENTS 
RESULTING  FROM  TRAINER 


Measure 

Mean  Value 
Class  407 
Group 

,  Mean  Value, 
Class  416 
Group 

Percent 

Improve¬ 

ment 

i 

Chance 

Difference 

Probability 

Number  of  No.  2 

Needle  Reading  Errors 

.58 

.09 

84% 

.001 

Number  of  Altimeter 
Reading  Errors 

.81 

.30 

63% 

.  01 

Number  of 

Wrong  Headings 

2.01 

.88 

56% 

*  .  001 

Number  of  Fuel 

Mgt.  Errors 

.58 

.36 

33% 

.05 

Minutes  to  Get 
on  Radial 

3.85 

2.47 

36% 

.01 

Percent  of  Time 
on  Radial 

59.65 

78.37 

31% 

.001 

Number  of  Missed 

Calls 

2.38 

1.65 

31% 

.05 

l 

ETAs,  No.  Minutes 

Off 

1.52 

1.08 

29% 

.01 

Number  of  Voice 

Comm.  Errors 

3.77 

2.86 

24% 

.05 

Number  of  Turn 

Point  Errors 

1.28 

1.06 

17% 

.05 
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TABLE  7.  SUMMARY  OF  MEASURES  SHOWING 

NO  DIFFERENCES  BETWEEN  GROUPS 


Measure 

Mean  Value 
Class  407 
Group 

Mean  Value 
Class  416 
Group 

Percent 

Change 

Frequency  of 

Departure  Errors  . 52 

.42 

19% 

Frequency  of 
Enroute  Errors 

.48 

.75 

56% 

Frequency  of 
Approach  Errors 

.72 

.84 

17% 

INSTRUCTOR  QUESTIONNAIRE  RESPONSES.  Questionnaires  were 
issued  to  instructor  NFOs  and  instructor  pilots  at  the 
conclusion  of  the  data  collection  period.  Content  of  the 
questionnaire  is  contained  in  Appendix  B.  Nine  of  the 
instructor  NFOs  and  five  instructor  pilots  completed 
portions  of  the  questionnaire  which  were  applicable  to 
transfer  of  training  from  Device  1D23  to  airborne  point-to- 
point  and  airways  navigation  and  communication  tasks. 

Responses  of  both  NFOs  and  pilots  were  quite  similar. 
Accordingly,  responses  from  both  groups  were  combined 
and  are  summarized  below. 

To  explore  the  general  feeling  of  instructors  regarding 
the  value  of  training  in  the  device,  they  were  asked  to 
rate  how  much  value  they  felt  training  in  Device  1D23 
had  been  in  preparing  students  for  each  T-39  training 
flight.  The  rating  scale  used  and  the  percents  of  instructors 
responding  to  each  category  are  presented  in  Table  8. 

A  majority  of  instructors  felt  that  training  in  the  device 
was  of  at  least  moderate  value. 

A  second  item  was  designed  to  examine  instructor  views 
in  greater  detail.  The  item  consisted  of  38  task  activities 
required  of  students  during  the  T-39  training  flights. 

The  activities  were  clustered  under  five  main  topic  areas. 
Instructors  were  asked  to  rate  how  well  they  felt  training 
in  the  device  prepared  students  to  execute  each  of  the  task 
activites. 
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TABLE  S.  INSTRUCTOR  RATINGS  OF  VALUE  OF  TRAINING 
FOR  T-39  FLIGHTS 


1 - 

Flights 

Rating  Scale  Values 

Negative 

None 

Minimal 

Moderate 

High 

1 

0% 

0% 

15% 

62% 

23% 

2 

0% 

0% 

8% 

69% 

23% 

3 

0% 

0% 

23% 

69% 

3% 

Table  9  summarizes  results  of  the  ratings.  An  "X" 
in  the  columns  of  the  table  shows  median  instructor  ratings. 
It  is  notable  that  none  of  the  consensus  (median)  ratings 
fell  within  the  categories  of  hurts  greatly  or  hurts  some. 
The  performance  of  nine  task  activities,  however,  were  rated 
as  having  been  unaffected  by  training  in  the  device.  The 
majority  of  these  activities  fell  under  the  general  topic 
area  of  copilot  duties;  very  few  task  activities  associated 
with  copilot  duties  can  be  trained  in  the  device.  Only 
two  task  activities,  determining  estimated  time  of  arrival 
and  interpreting  the  Radio  Magnetic  Indicator  were  rated 
as  having  been  helped  greatly  by  training  in  the  device. 

A  much  larger  number  of  activities  was  rated  in  the 
category  of  Helps  Some. 


TABLE  9.  SUMMARY  OF  TASK  ACTIVITY  RATINGS,  T-39  PHASE 


PREFLIGHT  PLANNING  &  PREPARATION 
Interpreting  airways  charts  .  , 
Selecting  radio  nav  aids  .  .  . 

Preflighting  logs  . 

Preparing  flight  plans  .  .  .  . 
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MFASURING  AND  COMPUTING 

Determining  TAS  ...  . 

Determining  ground  speed  ..... 

Determining  ETA  . 

Determining  wind  direction/velocity 
Determining  drift  angle  . 


X 

X 

X 

X 
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TABLE  9.  SUMMARY  OF  TASK  ACTIVITY  RATINGS r  T-39  PHASE  (continued) 


NAVIGATING 

Understanding  radials  from  nav  aids 

Interpreting  RMI  . 

Correctly  applying  drift  angle  .  . 
Determining  headings  to  fly  .  .  .  . 

Departure  procedures  . 

Enroute  procedures  . 

Turn  point  procedures  .  . 

Directing  flight  in  holding  pattern 

Approach  procedures  . 

Fuel  management  . 

Filling  out  logs  . 

Communicating  with  the  pilot  ... 
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COPILOT  DUTIES 

Interpreting  flight  instruments 
Interpreting  engine  instruments 
Performing  checklist  items  .  . 

Tuning  radios  . 

Setting  IFF  codes  . 

Monitoring  UHF  radios  ..... 
Communicating  with  the  gr our  d  . 
Copying  clearances  . 


INTEGRATION  OF  KNOWLEDGE 

Getting  it  all  together  . 

Interpreting  instruction  . 

Keeping  oriented  xelative  to  aircraft  . 
position 

Keeping  oreinted  relative  to  direction 
to  checkpoints 

Pacing  tasks  to  "keep  ahead  of  the  .  . 
aircraft" 

Anticipating  what  occurs  in  flight  .  . 

Understanding  spatial  relationships  .  . 

Identifying  incorrect  Nav  inputs/  .  , 
solutions 

Identifying  procedural  &  computation 
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Instructors  were  asked  to  elaborate  upon  why  they  felt 
additional  training  in  the  device  should  not  be  substituted 
for  T-39  training.  One  strong  response  pattern  addressed 
"the  realism  of  flight"  as  previously  discussed  in  Section 
III.  Additionally,  time  pressures  associated  with  jet 
flight  (420  KTAS  in  the  T-39)  were  cited.  A  second  strong 
response  pattern  centered  upon  the  need  for  voice 
communications  practice  in  an  airborne  environment,  which 
includes  static  and  high  communication  traffic  levels. 

Other  factors  cited  were:  opportunity  in  the  T-39 
for  students  to  learn  actual  rather  than  generalized  cock¬ 
pit  procedures;  experience  the  use  of  actual  flight 
instruments  for  flight  control,  engine  monitoring  and 
navigation;  and  practice  in  departure,  enroute,  checkpoint, 
and  approach  procedures  in  the  context  of  real-time  constraints. 

Instructors  also  were  asked  to  suggest  modifications 
to  the  trainer  or  TPs  which  might  make  it  possible  to 
substitute  training  in  the  device  for  in-flight  training  in 
the  T-39.  The  strongest  response  pattern  centered  upon 
modifying  radio  communications  in  the  device  to  include 
static  and  communications  patter  more  like  that  found  in 
the  in-flight  environment.  Additional  suggestions  were: 
facilitate  real-time  communication  in  the  device  by 
shortening  communication  queues;  implement  improved 
methods  for  monitoring,  evaluating  and  correcting  student 
communications;  improve  the  student-to-instructor  ratio; 
replace  the  digital  readouts  in  the  device  with  analog 
displays;  Diace  greater  training  emphasis  on  fuel  manage¬ 
ment;  and  place  greaver  training  emphasis  on 
developing  speed  in  executing  airways  navigation 
and  communication  tasks. 

In  two  separate  questions,  instructors  were  asked  to 
separately  list  elements  of  training  in  the  device  which 
they  felt  had  the  best  value  or  the  least  value  in  preparing 
students  for  T-39  traininq  flights.  Many  elements 
frequently  cited  as  having  the  best  value  could  be  categorized 
under  the  heading  of  hands-on  practice  in  a  near  real-time 
environment,  with  resulting  benefits  to  the  students  in: 
radial  tracking;  point-to-point  navigation  procedures;  enroute, 
check  point  and  approach  procedures;  computational  speed 
and  accuracy;  and  making  wind  corrections.  Other  elements 
centered  around  pacing  to  keep  ahead  of  the  aircraft, 
understanding  and  organizing  task  priorities,  and 
learning  to  anticipate  what  would  haopen  next  because  of 
an  improved  understanding  of  the  sequential  events  required 
in  jet  airways  navigation. 
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The  factor  listed  as  having  least  training  value 
was  voice  communications  training.  Instructors  generally 
felt  that  the  training  was  providing  benefit  to  the 
student  in  terms  of  content  and  format  for  radio  transmissions. 
However,  more  realistic  (noisy  and  with  traffic)  communications 
training  was  strongly  recommended.  A  second  factor  listed  by 
approximately  one  third  of  the  instructors  as  having  least 
value  was  the  use  of  digital  readouts  in  the  device.  In  the 
T-39,  the  student  is  confronted  with  actual  aircraft  displays 
which  require  different  reading  and  interpretation  skills. 

One  instructor  also  pointed  out  that  in  the  trainer,  the  flight 
is  largely  paced  by  the  student,  not  a  pilot.  In  flight, 
pilots  (and  the  speed  of  the  aircraft)  are  key  pacing  factors; 

•  some  students  have  experienced  difficulty  in  making  the  trans¬ 
ition. 

Finally,  instructors  were  asked  to  rate  the  concept  of 
using  Device  1D23  to  train  all  students  to  at  least  the  same, 
standardized  performance  levels  prior  to  allowing  them  to 
advance  to  in-flight  training  in  the  T-39.  Response  options 
were:  very  undesirable,  possibly  undesirable,  no  opinion, 
possibly  desirable,  and  very  desirable.  Sixteen  percent 
(two  instructor  NFOs)  rated  the  concept  as  at  least  possibly 
undesirable.  An  additional  38  percent  rated  the  concept  as  possi¬ 
bly  desirable,  while  the  remaining  46  percent  rated  the  concept 
as  very  desirable.  Those  rating  the  concept  as  very  or  possibly 
undesirable  indicated  that  they  felt  training  in  the  device 
was  of  less  value  in  preparing  students  for  T-39  flights 
than  in  preparing  them  for  dead  reckoning  navigation  flights. 
Apparently,  therefore,  requiring  students  to  demonstrate 
certain  proficiency  levels  in  the  device  prior  to  T-39 
training  would,  in  their  view,  amount  to  an  unnecessary 
exercise  with  only  minimal  payoff. 

CONCLUSIONS 

Based  solely  upon  statistical  analysis  of  flight  grades, 
it  would  have  to  be  concluded  that  training  in  Device  1D23 
had  no  effect  upon  the  proficiency  with  which  students  were 
able  to  execute  point-to-point  and  airways  navigation  and 
communication  tasks  in  an  airborne  setting.  However,  as 
discussed  in  Section  III,  it  is  felt  that  reliable  conclu¬ 
sions  regarding  the  training  effectiveness  of  Device  1D23 
cannot  be  drawn  from  flight  grades  alone. 

Based  upon  statistical  analysis  of  supplemental  measures 
of  student  task  performance,  it  can  be  concluded  that 
training  in  Device  ir?3  has  resulted  in  numerous  improvements 
in  student  performam-v  luring  T-39  training  flights. 

Greatest  improvements  v  »re  found  in  the  following  areas: 

Accuracy  with  which  the  number  t.*o  needle  of  the  RMI  is 
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read,  accuracy  with  which  the  altimeter  is  read,  headings 
to  be  flown,  fuel  management,  and  time  required  to  get  on 
radials.  Moderate  improvements  were  found  for:  the  percent 
of  time  students  were  able  to  stay  on  radials,  identifying  in¬ 
coming  radio  calls,  accuracies  of  ETAs,  and  voice  communica¬ 
tion  transmissions.  A  slight  improvement  also  was  found  in 
turn  point  procedures.  Overall,  performance  in  the  above 
areas  improved  by  an  average  of  41  percent. 

Three  student  performance  area-*  showed  no  statistically 
reliable  change  as  a  result  of  the  introduction  of  training 
in  Device  1D23.  It  can  be  concluded,  therefore,  that 
training  in  the  device  had  no  effect  upon  student  performance 
of  departure  procedures,  enroute  procedures  or  approach 
procedures . 

Task  areas  for  which  moderate  or  no  improvement 
in  student  performance  was  observed  following  training  in 
Device  1D23  can  be  considered  as  definitions  of  areas  in 
which  the  training  effectiveness  of  the  device  could  be 
further  enhanced. 

Instructor  opinion  also  supports  the  conclusion  that 
training  in  the  device  has  result*^  in  improvements  in  many 
aspects  of  point-to-point  and  airways  navigation  performance. 
It  was  instructor  consensus,  however,  that  the  training 
effectiveness  of  Device  1D23  is  greater  in  preparing  students 
to  execute  dead  reckoning  navigation  tasks  than  in  training 
them  to  execute  tasks  required  in  T-39  training  flights. 
Student  performance  data  collected  during  the  study  did  not 
allow  for  a  quantitative  verification  of  this  consensus  view. 

It  also  appears  reasonable  to  conclude  that  the 
effectiveness  of  the  device  in  preparing  students  for  T-39 
training  flights  could  be  improved  if  the  device  simulated 
noisier  and  more  heavily  trafficked  radio  communications , 
if  student- to- instructor  ratios  were  improved  to  provide 
better  control  over  communications  and  procedures  training, 
and  if  the  devices'  automated  performance  measurement 
capability  was  improved  to  give  a  better  description  of 
student  performance.  It  also  appears  reasonable  to  conclude 
that  the  effectiveness  of  the  device  could  be  *  ’.rther 
improved  if  it  were  used  to  establish  acceptable  levels 
of  student  performance  before  students  would  be  allowed 
to  advance  to  in-flight  training. 
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SECTION  V 

EFFECTS  OF  ADDITIONAL  TRAINING  IN  DEVICE  1D23 
UPON  DEAD  RECKONING  NAVIGATION 


OBJECTIVE 

The  objective  for  this  aspect  of  the  study  was  to 
determine  the  impacts  of  increased  training  in  Device  1D23 
upon  student  dead  reckoning  navigation  task  performance 
as  demonstrated  during  in-flight  training  in  the  T-29  or 
0114  aircraft. 

! 

CONTEXT 

Objectives  associated  with  in-flight  dead  reckoning  i 

navigation  training  and  the  content  of  in-flight  training 
were  the  same  as  previously  described  in  Section  III,  with 
one  exception.  The  number  of  dead  reckoning  training 
flights  had  been  reduced  from  five  to  four.  One  of  the 
first  two  flights  was  graded;  both  the  third  and  fourth 
flights  were  graded. 

The  trainer  was  kept  available  approximately  four  hours  { 

per  day.  In  keeping  with  standard  practices  in  effect  at 
VT-10  at  the  time  of  the  study,  students  were  allowed 
to  voluntarily  come  in  for  additional  training  in  the 
device  at  their  own  discretion. 

METHOD 

Class  421  was  selected  for  use  in  this  aspect  of  the 
study.  The  class  was  selected  because  it  was  available 
within  the  timeframe  of  the  study  and  because  instructor 
NFOs  indicated  that  students  in  the  class  were  representative 
of  contemporary  students  at  VT-10. 

Two  groups  of  15  students  each  were  selected  from  class 
421.  One  group  was  designated  the  control  group  and 
received  the  standard  amount  of  training  in  the  device.  The 

n  ^  ^  ..  ■  ■  4  .  —  JL  ^  ^ 
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received  one  additional  trainer  session. 

The  groups  were  balanced  on  four  dimensions.  First, 
each  group  contained  12  normally  matriculating  students 
plus  three  students  who  had  been  rolled  back  to  class  421  from 
preceeding  classes.  Second,  both  groups  contained  equal 
numbers  of  students  who  were  experiencing  academic  problems, 
as  indicated  by  low  grades  in  academic  subjects.  Both 
groups  also  were  highly  similar  in  terras  of  FAR  and  AQT 
scores.  For  both  groups,  FAR  stanine  scores  ranged  from 
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1  through  8,  with  a  mean  of  5.1.  For  the  control  group, 

AQT  st&nine  scores  ranged  from  4  through  9,  with  a  mean  of 
6.1.  For  the  experimental  group,  AQT  stanine  scores  ragged 
from  3  through  9  with  a  mean  of  5.8. 

Both  groups  of  students  received  introductory  training 
in  the  operation  of  Device  1D23  (TPs  1  and  2) ,  followed 
by  proscribed  training  on  TPs  3,4,5,  and  6 .  Experimental 
group  students  received  one  additional  training  session  on 
TP  4.  The  additional  session  was  administered  on  the  day 
after  prescribed  training  on  TP  6  had  been  completed  for 
both  groups. 

TP  4  was  selected  for  the  additional  training  because 
it  was  designed  to  emulate  the  lead-tracker  navigator 
structure  of  the  dead  reckoning  training  flights. 

From  a  Thorndikeian  viewpoint,  transfer  of  training  is 
maximized  when  elements  of  training  tasks  and  transfer 
tasks  are  identical.  In  this  context,  TP  4  appear  to  offer 
the  best  opportunity  within  the  constraints,  of  the  study 
to  determine  whether  additional  training  in  the  device 
would  have  any  in-flight  payoff  for  dead  reckoning  tasks. 

A  sample  of  instructor  NFOs  also  agreed  that  TP  4  offered  the 
highest  potential  for  payoff. 

Each  dead  reckoning  training  flight  consisted  of 
approximately  equal  numbers  of  students  from  the  experimental 
and  control  groups.  This  was  done  in  part  so  that  students 
in  both  groups  would  be  graded  and  otherwise  evaluated 
by  the  same  instructor  NFOs.  Instructors  were  not  told 
which  students  were  in  each  group,  although  they  were  aware 
that  some  of  the  students  had  received  additional  training. 

It  also  was  done  to  equate  in-flight  environmental  factors 
for  both  groups. 

Students  in  both  groups  were  graded  after  each  flight 
using  standardized  Aviation  Training  Forms  and  grading 
criteria.  The  four  point  scale  previously  described  in 
Section  III  was  used  to  determine  a  grade  for  each  performance 
element  which  was  itemized  in  the  Aviation  Training  Form  and 
which  was  applicable  to  the  student's  requirements  during 
the  flight.  An  overall  flight  grade  was  computed  for 
each  student  in  both  groups  for  each  flight.  This  was  done  by 
averaging  numerical  grades  across  all  performance  elements. 

Following  each  flight,  students  in  both  groups 
were  required  to  complete  a  student  questionnaire,  which 
actually  was  a  self  evaluation  form.  It  was  intended 
to  allow  each  student  to  express  an  overall  perception  of  his 
performance  on  the  flight  and  a  projection  of  his  performance 
on  the  next  flight.  The  form  also  allowed  each  student  to 
express  levels  of  confidence  he  felt  in  executing  selected 
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tasks.  Content  of  the  student  questionnaire  is  contained  in 
Appendix  D. 

Following  each  flight,  instructors  were  required  to 
complete  a  student  critique  form.  The  form  was  designed 
to  supplement  normal  in-flight  grades  by  requiring 
instructors  to  express  their  overall  perceptions  of  student 
performance  and  to  list  the  occurrence  of  selected 
student  behaviors.  Content  of  the  student  critique  form 
also  is  contained  in  Appendix  D. 

Flight  grades  and  student  and  instructor  overall 
ratings  of  student  performance  were  statistically  analyzed. 
Multiple  discriminant  analysis  was  used  to  compare  the 
experimental  and  control  groups  on  a  composite  set  of 
measures  consisting  of:  flight  grades  for  the  first 
graded  flight,  the  third  flight,  and  the  fourth  flight? 
student  s.  If  ratings  of  performance  on  each  flight;  student 
self  ratings  of  anticipated  performance  on  the  next 
flight;  instructor  ratings  of  student  performance  on  each 
flight;  and  instructor  ratings  of  anticipated  student 
performance  on  the  next  flight.  Factorial  analysis  of 
variance  also  was  used  to  compare  the  experimental  and 
control  groups  separately  on  flight  grades  and  performance 
ratings . 

Other  data  developed  from  the  content  of  student 
post-flight  questionnaires  and  instructor  critique  forms 
were  separately  analyzed  to  compare  experimental  and 
control  groups. 

RESULTS 

An  examination  of  records  of  students  voluntarily 
seeking  extra  training  in  the  device  during  free  hours 
showed  that  approximately  equal  numbers  of  students 
in  each  group  had  come  in  for  voluntary  extra  training. 
Accordingly,  this  factor  should  not  have  differentially 
affected  group  performance. 

None  of  the  statistical  tests  showed  any  significant 
differences  between  the  experimental  and  control  groups. 

It  can  be  reasonably  concluded,  therefore,  that  any  minor 
variations  between  groups  were  due  to  chance  factors 
rather  than  to  effects  of  the  additional  training  in 
Device  1D23. 

Results  o*  the  statistical  analyses  were  confirmed  by 
instructor  comments.  In  interviews  conducted  during  and 
after  in-flight  training,  a  sample  of  instructor  NFOs 
indicated  that,  on  the  basis  of  their  observations  of 
student  performance,  they  were  unable  to  identify  which 
students  had  received  additional  training  in  the  device. 
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In  a  different  vein,  was  found  that  student  self- 
ratings  of  performance  averaged  approximately  one  scale 
unit  higher  for  both  groups  than  the  same  ratings  made  by 
instructors.  The  difference  was  statistically  significant 
at  the  .01  level. 

CONCLUSION 

Based  upon  results  of  the  statistical  analyses  and 
upon  instructor  comments,  it  can  be  reasonably  concluded 
that  one  additional  training  exposure  to  TP  4  had  no  effect 
upon  student  in-flight  performance  of  dead  reckoning  navigation 
tasks. 
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SECTION  VI 

EFFECTS  OF  ADDITIONAL  TRAINING  IN  DEVICE  1D23 
UPON  AIRWAYS  NAVIGATION 


OBJECTIVE 

The  objective  for  this  aspect  of  the  study  was  to 
determine  the  impacts  of  increased  training  in  Device  1D23 
upon  student  airways  navigation  and  communication  task 
performance  as  demonstrated  during  in-flight  training  in 
the  T-39  aircraft* 

CONTEXT 

Airways  navigation  and  communication  training  in 
Device  1D23  and  in  the  T-39  aircraft  were  the  same  as 
previously  described  in  Section  IV,  with  two  exceptions. 

A  new  training  problem  (TP  12)  was  developed  by  VT-10 
personnel  to  provide  expanded  training  in  the  device  in 
point-tc-point  navigation.  The  new  TP  was  added  to  the 
pre-T-39  trainer  syllabus.  Additionally,  the  first 
T-39  flight  was  changed  from  a  point-to-point  navigation 
flight  to  an  airways  navigation  flight,  thus  making 
all  three  T-39  flights  airways  training  flights.  All 
three  flights  were  graded. 

The  trainer  was  kept  available  approximately  four  hours 
per  day.  In  keepinq  with  standard  practices  in  effect  at 
VT-10  at  the  time  «f  study,  students  were  allowed  to 
voluntarily  come  in  for  additional  training  in  the  device 
at  their  own  discretion. 

METHOD 

Class  418  was  selected  for  use  in  this  aspect  of  the 
study.  The  class  was  selected  because  it  was  available 
within  the  timeframe  of  the  study  and  because  instructor 
NFOs  indicated  that  students  in  the  class  were  representative 
of  contemporary  students  at  VT-10. 


Two  groups  of  14  students  each  wore  selected  from 
class  418.  One  group  was  designated  the  control  group  and 
received  the  standard  amount  of  training  in  the  device. 

The  other  group  was  designated  the  experimental  group  and 
received  one  additional  trainer  session. 

The  groups  were  balanced  on  four  dimensions.  First, 
each  group  contained  12  normally  matriculating  students 
plus  two  students  who  had  been  rolled  back  to  class  418 
from  preceeding  classes.  Second,  both  groups  contained 
equal  numbers  of  students  who  were  experiencing  academic 
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problems.  Both  groups  also  were  similar  in  terms  of  FAR  and 
AQT  scores.  FAR  stanine  scores  ranged  from  1  through  9  for 
both  groups.  Mean  FAR  score  for  the  control  group  was  5.4, 
while  mean  FAR  score  for  the  experimental  group  was  5.5. 

AQT  stanine  scores  for  the  control  group  ranged  from  3 
through  9,  with  a  mean  of  6.6.  AQT  stanine  scores  for  the 
experimental  group  ranged  from  4  through  8,  with  a  mean 
of  6.1. 

Both  groups  of  students  had  received  training  in  the 
device  on  TPs  1  through  6,  followed  by  in-flight  dead 
reckoning  navigation  training.  Both  groups  also  received 
prescribed  training  on  TPs  7,  8,  10,  and  12.  Experimental 
group  students  received  one  additional  training  session 
on  TP  7.  The  additional  session  was  administered  on 
the  day  after  prescribed  training  on  TP  12  had  been 
completed  for  both  groups. 

The  selection  of  TP  7  for  the  additional  training  of 
experimental  group  students  was  done  in  concert  with  a 
sample  of  instructor  NFOs.  It  was  selected  because  it  was 
designed  to  exercise  the  student  in  airways  navigation  and 
IFR  voice  communications  requirements  under  nonradar  contact 
conditions.  In  these  respects,  task  requirements  were 
similar  with  those  required  during  T-39  airways  navigation 
training  flights.  Accordingly,  TP  7  appeared  to  offer 
the  best  opportunity  within  the  constraints  of  the  study 
to  determine  whether  additional  training  in  the  device 
would  have  any  in-flight  payoff  for  airways  navigation  and 
communication  tasks. 

Each  T-39  training  flight  contained  students  from  the 
two  groups.  In  this  way,  students  in  both  groups  were 
graded  and  otherwise  evaluated  by  the  same  instructor 
NFOs  and  pilots.  Instructors  were  not  told  which  students 
were  in  each  group,  although  they  were  aware  that  some  of 
the  students  had  received  additional  training.  Including 
students  from  both  groups  in  each  flight  also  equated  in-flight 
environmental  factors  for  both  groups. 

While  in  flight,  the  performance  of  each  student  was 
evaluated  by  instructors  using  the  supplementary  performance 
evaluation  forms  previously  described  in  Section  IV. 

Students  in  both  groups  were  graded  after  each  flight 
using  standardized  Aviation  Training  Forms  and  grading 
criteria.  The  four  point  scale,  previously  described  in 
Section  III,  was  used  to  determine  a  grade  for  each  performance 
element  which  was  itemized  in  the  Aviation  Training  Form  and 
which  wa3  applicable  to  the  student’s  requirements  during 
the  flight.  An  overall  flight  grade  was  computed  for  each 
student  in  both  groups  for  each  flight.  This  was  done  by 
averaging  numerical  grades  across  all  performance  elements. 
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Following  each  flight,  students  in  both  groups  were 
required  to  complete  the  student  questionnaire  described  in 
Appendix  D.  Following  each  flight,  instructors  were 
required  to  complete  a  student  critique  form, which  also 
is  described  in  Appendix  D.  Flight  grades,  student  and 
instructor  overall  ratings  of  student  performance,  and 
supplemental  performance  data  were  statistically  analyzed. 
Multiple  discriminant  analysis  was  used  to  compare  the 
experimental  and  control  groups  on  a  composite  set  of 
measures  consisting  of:  flight  grades  for  each  flight; 
student-self  ratings  of  performance  on  each  flight; 
student- self  ratings  of  anticipated  performance  on  the 
next  flight;  instructor  ratings  of  student  performance 
on  each  flight;  and  instructor  ratings  of  anticipated  student 
performance  on  each  flight.  Factorial  analysis  of  variance 
also  was  used  to  compare  the  experimental  and  control  groups 
separately  on  flight  grades  and  performance  ratings. 

Multiple  discriminant  analysis  also  was  used  to 
compare  the  two  groups  on  a  composite  set  of  13  measures 
derived  from  the  supplemental  performance  evaluations. 
Factorial  analysis  of  variance  also  was  used  to  separately 
yompare  the  two  groups  on  each  of  the  13  measures. 

Other  data  developed  from  the  content  of  student 
post-flight  questionnaires  and  instructor  critique  forms 
were  separately  analyzed  to  compare  experimental  and 
control  groups . 

RESULTS 

An  examination  of  records  of  voluntary  extra 
training  in  the  device  during  the  time  when  class  418  was 
receiving  pre-T-39  training  showed  that  approximately 
equal  numbers  of  students  in  each  group  had  come  in  for 
voluntary  extra  training.  Accordingly,  this  factor  should 
not  have  differentially  affected  in-flight  performance  of 
the  two  groups. 

None  of  the  statistical  tests  showed  any  significant 
differences  between  the  experimental  and  control  groups. 

It  can  be  reasonably  concluded,  therefore,  thac  minor 
variations  between  groups  were  due  to  chance  factors 
rather  than  to  effects  of  the  additional  training  in 
Device  1D23. 

In  a  different  vein,  it  was  again  found  that  student 
self-ratings  of  performance  were  generally  higher  than 
ratings  by  instructors.  The  trend  was  different,  however, 
than  that  found  for  dead  reckoning  flights.  Following  the 
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first  two  airways  flights,  students  rated  their  performance 
approximately  one  half  a  scale  unit  higher  than  did 
instructors.  These  differences  were  statistically  significant 
at  the  .01  level.  Following  the  third  airways  flight,  however, 
student  and  instructor  perceptions  of  overall  student  per¬ 
formance  appear  to  have  coincided,  because  ratings  by 
instructors  and  students  were  statistically  comparable 
following  the  third  flight. 

CONCLUSION 

Based  upon  results  of  the  statistical  analyses 
it  can  be  reasonably  concluded  that  one  additional  training 
exposure  to  TP  7  had  no  effect  upon  student  in-flight 
performance  of  airways  navigation  and  communication  tasks. 

This  conclusion  is  supported  by  instructor  comments. 
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SECTION  VII 

EVALUATION  OP  SELECTED  DEVICE  DESIGN 
FEATURES 

INTRODUCTION 

An  important  element  of  any  training  effectiveness 
evaluation  is  the  development  of  design  feedback  information. 
The  feedback  should  be  of  the  type  which  will  be  of  value 
to  design  teams  in  specifying  and  developing  similar 
training  devices  in  the  future.  Accordingly,  design  feedback 
information  should  include  descriptions  of  features  which 
fell  short  of  expectations,  along  with  explanatory  information 
regarding  why  they  may  have  fallen  short.  This  type  of 
feedback  identifies  potential  pitfalls  to  be  avoided  in 
future  designs.  Design  feedback  information  also  should 
identify  features  which  appeared  to  meet  expectations, 
because  such  information  provides  positive  guidance 
for  future  designs.  The  first  type  of  design  feedback  is 
much  easier  to  provide  than  is  the  second. 

A  device  as  large,  complex/ and  multi-faceted  as  Device 
1D23  incorporates  a  great  number  of  design  features.  Simply 
identifying  a  comprehensive  set  of  all  features  could  be 
a  major  undertaking.  Objectively  evaluating  each  feature 
would  require  even  greater  effort  which ,  ideally,  should 
involve  the  development  of  criteria  of  acceptibility  and 
a  determination  of  whether  each  feature  (or  selected 
combinations  of  features)  met  criterion.  The  criteria  of 
acceptibility  should  be  based  not  only  upon  desired 
levels  of  performance  in  the  user  environment,  but  also 
upon  any  cost-effectiveness  tradeoffs  which  may  have  been 
made  during  the  specification  and  development  of  the  device. 
The  advantage  of  identifying  and  objectively  evaluating 
each  feature  is  that  a  comprehensive,  unified  body  of 
design  feedback  information  can  be  generated.  A 
disadvantage  is  the  amount  of  resources  required  to 
generate  the  information. 

Selected  design  feedback  can  be  generated  in  a  much 
more  parsimonious  manner  by  addressing  features  which, 
through  use,  have  surfaced  as  being  notably  good  or  notably 
poor.  Four  problems,  however,  are  inherent  in  this  approach. 

A  primary  problem  is  that  it  is  possible  to  overlook 
notable  design  features.  This  can  occur  because 
a  comprehensive  set  of  all  features  is  not  used  to  provide 
guidance.  It  also  can  oc^ur  because  features  which  work 
quite  well  are  apt  to  became  inconspicuous  and  taken  for 
granted. 
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A  second  problem  is  that  uniform  criteria  for  evaluating 
design  features  may  not  be  applied.  It  is  particularly 
likely  that  cost-effectiveness  tradeoff  factors  may  not 
receive  formal  attention. 

A  third  problem  is  that  it  is  possible  for  resulting 
design  feedback  information  to  appear  more  negative  than 
might  actually  be  the  case.  This  follows  since  it  is  much 
easier  to  identify  design  features  which  have  been  problematic 
than  those  which  have  not.  Consequently,  resulting  design 
feedback  may  be  off  balance. 

Finally,  the  more  parsimonious  approach  places  an 
additional  burden  upon  teams  designing  future  training 
devices.  This  results  since,  essentially,  no  design 
feedback  may  be  provided  for  "low  profile"  features  which 
may  have  functioned  quite  acceptably.  The  burden  of  iden¬ 
tifying  such  features  is  placed  upon  members  of  the  design 
team. 


Problems  inherent  in  a  more  parsimonious  approach, 
however,  are  offset  to  some  degree  by  one  very  practical 
factor.  An  advantage  of  the  approach  is  the  enhanced 
likelihood  that  at  least  some  design  feedback  can  be 
provided  within  the  constraints  of  resources  which  are 
available  for  generating  the  feedback.  In  the  present  study, 
design  feedback  information  developed  only  for 
selected  device  design  features  because  of  .:escu>;c.i 
constraints.  Problems  inherent  in  the  parsimonious  approach, 
therefore,  are  reflected  in  the  design  feedback  which  is 
presented  in  this  section.  Specific  procedures  for  developing 
the  information  are  presented  below. 

METHOD 

Numerous  conversations  were  held  with  instructor  NFOs, 
Training  Device  Operators  (  TDs  )  and  device  maintenance 
personnel  throughout  the  planning  and  execution  phases 
of  the  study.  The  conversational  interviews  were  supplemented 
by  direct  observations  of  device  utilization.  The  observations 
frequently  involved  "show  me  what  you  mean  by  that"  demonstra¬ 
tions  of  relevance  to  various  design  features.  More  limited 
conversational  interviews  also  were  held  with  students  who 
were  receiving  training  in  the  device.  Extensive  notes 
were  taken  during  and  after  the  interviews. 

Based  upon  preliminary  interviews,  two  device  evaluation 
questionnaires  were  developed.  They  were  developed  to 
provide  a  structured  context  within  which  instructors  and 
students  could  express  comments  regarding  significant 
features  of  the  device.  Content  of  the  questionnaire 
which  was  administered  to  instructor  NFOs  is  contained  in 
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Appendix  A.  Only  instructors  who  were  qualified  to  administer 
training  in  the  device  completed  the  questionnaire. 

Accordingly,  questionnaire  responses  were  obtained  from  11  in¬ 
structors,  Content  of  the  questionnaire  which  was  administered 
to  students  is  shown  in  Appendix  E.  The  questionnaire 
was  administered  to  students  class  418  immediately 
following  their  inflight  training  in  the  T-39  aircraft. 

The  questionnaire  also  was  administered  to  students  in  class 
422  immediately  following  their  inflight  training  in 
dead  reckoning  navigation  (T-29  and  C-114  aircraft) .  A 
total  of  57  students  completed  questionnaires. 

Questionnaire  responses  were  combined  with  content  of 
the  notes  taken  throughout  the  planning  and  execution  of 
the  study.  The  resulting  information  was  then  organized 
into  main  topic  areas.  The  resulting  information  is 
presented  below. 

OVERALL  DESIGN  ACCEPTANCE 

Device  1D23  was  very  well  received  by  instructors,  students. 
Training  Device  Operators  (TDOs)  and  maintainers.  General 
praise  for  the  device  involved  both  physical  design  features 
and  the  ability  of  the  device  to  provide  meaningful  learning 
experiences.  In  these  respects,  it  would  appear  that  the 
device  reflects  a  general  design  philosophy  which  should 
have  meaningful  utility  in  the  design  of  other  multi-station 
trainers . 

FACILITY  DESIGN 

Colors,  sound  levels,  ventilation  and  illumination  were 
found  to  be  pleasant,  comfortable,  and  well  liked.  Overall 
walk-around  accessibility  to  operator  consoles,  student 
stations,  computers,  input-output  devices  and  storage 
files  were  quite  acceptable.  Some  student  queueing  was 
observed  at  the  performance  measurement  line  printer 
following  training  and  at  the  door  to  the  facility  during 
entry  and  exit  of  classes.  Both  types  of  gueues  were 
transient,  however. 

Tvm»  tit*  r>  Tr  rfmr 
i-xnJui  AnxiinDJ-iiAi  X 

The  device  was  quite  new  at  the  time  the  present  study 
was  conducted.  At  that  time,  however,  maintenance  of  the 
device  appeared  to  be  straightforward  and  well  in  hand. 
Maintenance  documentation  and  maintainer  training  appeared  to 
have  been  adequate. 

Access  to  the  computers  was  found  to  be  quite  acceptable. 
Accessibility  was  one  of  the  factors  which  facilitated 
program  loading  and  computer  maintenance.  Similarly,  all 


l 


47 


NAVTRAEQUIPCEN  72-C-0209-2 

console  control-display  units  were  readily  accessible 
for  maintenance. 

The  only  major  maintenance  problem  which  surfaced  during 
the  timeframe  of  the  study  involved  access  to  logic  circuitry 
at  the  front  (nearest  the  wall)  of  each  student  station. 

Accass  was  gained  through  a  Hatch  on  top  of  the  front  portion 
of  student  stations,  and  required  maintainers  to  stand  on 
the  sides  of  the  stations  and  lean  over  and  down  into  the 
logic  circuitry  areas.  Using  the  sides  of  student  stations 
as  foot  holds  appeared  less  than  desirable  from  a  structural 
standpoint.  Additionally,  the  required  procedure  appeared  to 
pose  a  safety  problem  for  maintainers. 

Because  of  low  ceiling  height  in  one  area  of  the 
trainer  room,  it  was  impossible  to  gain  access  (using  the 
leaning  procedure  described  above)  to  one  of  the  logic 
circuit  areas  of  one  of  the  student  stations.  When  the 
station  experiences  a  malfunction  of  certain  circuitry,  it 
is  likely  that  it  will  remain  in  a  down  state  for  quite 
a  while. 

Maintainability  of  other  elements  of  the  trainer  appeared 
to  pose  no  special  problems.  In  fact,  several  maintainers 
indicated  that,  by  and  large,  maintenance  of  the  device  was 
more  straightforward  than  they  had  experienced  on  other 
simulators  and  trainers  of  similar  size  and  complexity. 

SOFTWARE  DOCUMENTATION 

Programming  changes  to  TPs  and  creating  new  TPs 
involved  straightforward  programming  procedures.  Documentation 
of  procedures  for  changing  or  creating  TP  software,  in 
combination  with  instructor  training,  appeared  to  have  been 
sufficient. 

Operating  system  software,  however,  did  not  appear  to 
have  been  adequately  documented.  Many  changes  which  were 
felt  to  be  desirable  with  respect  to  the  operation  of  the 
trainer  could  not  be  made  by  Navy  personnel.  This  resulted 
because  documentation  which  accompanied  the  device  did  not 
describe  the  system  software  in  sufficient  detail. 

Other  documentation  and  user  training  on  device 
operation  and  maintenance  appeared  to  have  been  adequate  in 
most  cases. 

INSTRUCTOR  AND  OPERATOR  CONSOLES 

Instructor  and  TDO  consoles  were  found  to  be  well 
organized  and  well  accepted.  Chairs  which  were  provided 
were  very  comfortable.  CRT  displays  were  "flicker  free". 
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Alphanumerics  and  symbology  were  legible  under  the  ambient 
illumination  levels  in  the  trainer  room.  The  layout  of  console 
keyboards  was  found  to  be  quite  workable  by  instructors  and 
TDOs.  Instructors  did  express  a  desire,  however,  for  a  switch 
to  deactivate  their  microphones.  Presently,  their  microphones 
are  always  hot,  and  accidental  communication  to  students  is 
possible.  To  avoid  this,  microphones  had  to  be  unplugged 
when  not  being  intentionally  used.  Additionally,  longer 
microphone  cords  were  desired  by  several  instructors  in  order 
to  allow  them  to  walk  around  and  observe  students  while  still 
being  able  to  use  the  intercom  systems. 

Information  display  modes  which  could  be  selected  for 
on-line  use  at  instructor  and  TDO  consoles  were  found  to  be 
generally  very  usable.  Instructors  made  a  number  of  recom¬ 
mendations,  however,  for  enhancing  displayed  information, 
simplifying  control  operations,  and  enhancing  the  training 
utility  of  the  device.  The  recommendations,  summarized  below, 
primarily  involved  software  design.  Many  appear  to  have 
generality  to  other  multi-station  training  devices. 

Uniform  capability  for  both  group  and  individual  student 
station  control  was  highly  recommended.  Effecting  group 
changes  by  changing  values  for  individual  stations  frequently 
can  require  hundreds  of  keyboard  entries,  which  take  time  and 
raise  the  likelihood  of  a  data  entry  error.  Presently,  some 
control  functions,  such  as  restarting  stations,  can  be 
accomplished  only  for  units  of  20  stations.  Individual  re¬ 
start  capability  would  be  very  useful  for  allowing  individual 
students  to  start  over.  A  group  takeoff  permissive  command 
capability  would  be  desirable  to  complement  the  present 
individual  takeoff  permissive  capability.  Similarly,  a  group 
ability  to  make  changes  in  programmed  system  values,  such  as 
winds  and  system  malfunctions,  also  was  desired. 

The  on-line  inspect  and  change  mode  was  found  to  be  quite 
valuable  for  entering  or  overriding  factors  such  as  atmospheric 
data,  malfunctions,  and  annunciator  messages.  Instructors  felt 
that  the  on-line  ability  to  change  navigation  aids  should  have 
been  included  in  the  mode.  Again,  both  group  and  individual 
student  station  control  was  desired  for  this  mode. 

The  inclusion  of  an  on-l/ne  capability  to  segregate 
individual  student  stations  from  elements  of  the  systems' 
software  would  have  been  valuable.  Student  stations  which 
were  not  used  for  training  had  to  be  brought  on  line  to  provide 
for  shifting  a  student  to  another  station  if  his  malfunctioned 
or  to  have  a  station  ready  for  a  student  to  use  to  independent¬ 
ly  practice  navigation  tasks.  This  was  necessary  because 
stations  which  were  to  be  active  had  to  be  designated  during 
system  initialization.  It  was  not  possible,  therefore,  to 
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activate  a  station  after  a  TP  had  begun  without  stopping 
training,  reinitializing  ,  and  starting  the  TP  from  the 
beginning. 

The  presence  of  unused  but  activated  stations  which  were 
not  segregated  from  stations  being  used  had  several  effects. 

The  graphic  display  of  student  positions  on  console  CRT 
displays  was  automatically  centered  based  upon  an  average 
weighting  of  the  geographic  positions  of  all  activated  student 
stations.  Those  which  were  active  but  which  were  still  Mon  the 
ground"  resulted  in  a  biased  centering  of  the  graphic  display. 
Occasionally,  students  would  "fly  off  the  display"  because  of 
the  biased  centering.  As  a  result,  the  use  of  manual  centering 
procedures  frequently  was  required. 

When  students  ca.'e  in  for  extra  unstructured  training, 
while  formal  training  was  ongoing,  it  was  necessary  to  execute 
a  tedious  procedure  to  free  a  station  from  the  TP  which  was 
being  used.  The  procedure  involved  getting  the  station 
airborne,  entering  a  short  ETA,  allowing  the  ETA  to  expire. 
Unfreezing  the  station,  repositioning  it  onto  the  next  leg,  and 
repeating  the  procedure  until  all  legs  had  been  accounted  for. 
The  repositioning  procedure  also  was  slow  and  tedious.  It 
would  have  been  far  more  desirable  simply  to  designate  an 
active  station  to  be  segregated  from  the  display  centering  and 
performance  measurement  software. 

Finally,  active  stations  which  were  not  being  used  were 
scored  by  the  device's  automated  performance  measurement 
system  as  having  made  no  errors.  The  contribution  of  this 
erroneous  data  to  student  performance  measurement  is  discussed 
more  fully  in  a  subsequent  section.  Again,  hov^ever,  it  would 
have  been  highly  desirable  simply  to  designate  an  active 
station  to  be  segregated  from  the  performance  measurement 
software. 

The  graphic  display  of  student  station  positions  relative 
to  the  course  which  was  to  be  flown  was  felt  to  be  quite 
valuable,  particularly  for  monitoring  overall  group  performance 
and  identifying  students  who  were  not  adhering  to  the  flight 
plan.  Several  improvements  in  the  display  were  suggested, 
however.  Small  lines  showing  the  direction  of  the  velocity 
vector  of  each  station  could  be  displayed.  The  vectors  proved 
to  be  of  considerable  valve  in  determining  whether,  for 
example,  a  station  which  was  jff  course  was  converging  on  or 
divergina  from  the  course.  It  was  recommended  that  a  simple 
control  action  to  produce  vectors  for  all  stations  on  any  of 
the  CRT  displays  would  be  quite  desirable.  Additionally,  it 
was  suggested  that  vectors  for  all  stations  be  displayed  when 
the  SACS  (specific  aircrart  submode)  mode  was  selected.  The 
SACS  mode  was  used  to  obtain  a  display  of  selected  performance 
data  for  a  designated  student  station. 
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On  the  graphics  display,  up  to  20  pips  were  displayed 
simultaneously,  one  for  each  activated  student  station.  No 
identification  was  displayed  regarding  which  pip  was  associated 
with  which  student  station.  Making  the  identification  required 
the  instructors  to  execute  a  search  routine  using  the  SACS 
mode.  With  this  mode,  a  special  "S"  symbol  appeared  over  the 
pip  which  had  been  designated  by  entry  of  a  station  number. 

Up  to  20  SACS  searches  were  required  to  identify  a  particular 
pip.  An  improved  means  for  identifying  pips  would  be  highly 
desirable. 

The  special  "S'*  symbol  used  with  the  SACS  mode  was  found 
to  be  quite  useful.  It  was  recommended  that  the  symbol  appear 
over  the  pip  of  a  student  station  when  instructors  call  the 
station  using  the  intercom  system.  Additionally,  it  was 
recommended  that  the  SACS  selected  performance  data  be 
displayed  for  the  station  being  addressed.  Additionally,  it 
was  recommended  that  just  the  symbol  be  displayed  on  TDO 
consoles  during  communications  between  TDOs  and  students. 
Presently,  neither  the  symbol  nor  the  SACS  information  is 
displayed  automatically  during  communications. 

The  out-of-tolerance  (AOTS)  display  mode  was  found  to  have 
little  practical  value.  The  mode  could  either  automatically 
or  upon  command  display  selected  performance  data  for  stations 
which  were  outside  cf  programmed  tolerance  limits.  One 
problem  with  using  the  mode  was  that,  typically,  several 
students  would  be  out- of- tolerance  simultaneously.  The 
ambiguity  problem  of  which  pip  represented  which  station,  as 
discussed  above,  rendered  the  mode  of  little  value.  Additionally, 
more  expanded  performance  data  could  be  obtained  through  the  SACS  mode. 


The  addition  of  an  auditory  or  visual  signal  to  alert 
instructors  that  students  were  attempting  to  communicate  with 
them  over  the  intercom  also  was  recommended. 

Finally,  software  which  was  designed  to  freeze  a  student 
station  if  it  exceeded  programmed  limits  and  alert  the 
instructor  of  the  freeze  was  circumvented.  It  was  found, 
particularly  during  early  TPs,  that  numerous  student  stations 
were  being  simultaneously  frozen.  Because  of  the  student- to- 
instructor  ratios  which  were  common,  instructors  could  not 
attend  rapidly  enough  to  all  stations.  Accordingly,  the 
stations  were  simply  unfrozen  and  allowed  to  resume  navigation. 
Additionally,  instructors  developed  the  viewpoint  that  it  was 
not  desirable  to  freeze  stations.  Rather,  it  was  better  to 
walk  over  to  an  out- of- tolerance  station  and  work  with  the 
student  ir,  real  time  to  identify  and  resolve  his  navig-*  tional 
errors. 
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STUDENT  STATIONS 

Student  stations  were  generally  well  received  by  both 
students  and  instructors.  The  total  volume  of  each  station 
was  adequate  to  accommodate  students  without  producing 
claustrophobic  responses.  However,  the  relatively 
small  size  of  the  stations  made  it  difficult  for  instructors 
to  3ee  student  logs  and  charts  to  diagnose  learning 
problems.  Arrangements  of  navigation,  communication  and 
computer  entry  controls  and  disDlays  appeared  to  be  orderly 
and  logically  organized.  All  controls  and  displays  were 
adequately  marked  and  labelled. 

Students  and  instructors  were  asked  to  rate  the 
relative  difficulty  which  students  experienced  in  acclimating 
to  several  aspects  of  the  student  stations.  Percents 
of  students  and  instructors  who  responded  to  each  difficulty 
category  are  summarized  in  cable  10  for  four  aspects  of  the 
student  stations. 

TABLE  10.  ESTIMATES  OF  STUDENT  DIFFICULTY  IN  ACCLIMATING 
TO  STUDENT  STATION 


Levels  of  Difficulty 
None  Little  Moderate  High 


Interpreting  Digital 
Readout  Displays 

58%* 

(33%)** 

30% 

(56%) 

7% 

(11%) 

5% 

(  0%) 

Using  Keyboards  for 

Data  Inputs 

30% 

(11%) 

49% 

(34%) 

16% 

(44%) 

5% 

(11%) 

Learning  Requirements 
and  Procedures  for 
Control  of  Trainer 

17% 

(  0%) 

46% 

(56%) 

32% 

(11%) 

5% 

(33%) 

Identifying  Uses  of 
Cockpit  Functional 

Areas 

23% 

(  0%) 

59% 

(33%) 

13% 

(56%) 

5% 

(11%) 

*  Percents  of  Students 
**  Percents  of  Instructors 


For  all  four  categories,  instructors  tended  to  indicate 
that  students  experienced  slightly  higher  difficulty  levels 
than  was  indicated  by  students.  In  no  case,  however,  did  a 
majority  of  either  students  or  instructors  indicate  that 
high  difficulty  was  experienced. 
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Student  stations  incorporated  many  light  emitting  diode 
(LED)  alphanumeric  readouts  for  aircraft  parameters, 
annunciator  messages,  and  computer  generated  data.  The 
readouts  were  sufficiently  legible,  although  circuits  within 
the  displays  were  visible.  Learning  to  read  and  interpret 
digitally  displayed  data  did  not  appear  to  pose  a  problem. 

In  questionnaire  responses,  numerous  students  indicated  that 
the  digital  readouts  were  one  of  the  aspects  which  they 
liked  best  about  the  student  stations.  However,  many 
instructors  and  students  also  indicated  a  preference  for 
broader  use  of  analog  displays,  particularly  for  altitude 
and  time  displays.  Many  students  expressed  favorable 
attitudes  toward  analog  displays  (attitude  and  RMI)  which 
were  incorporated  into  student  stations.  A  number  of  students 
and  instructors  also  indicated  a  need  for  the  addition  of 
a  capability  to  display  the  last  ETA  entered  into  the 
computer  by  a  student.  Entry  of  correct  ETAs  was  critical 
to  the  student's  achieving  a  high  score  from  the  trainer's 
automated  performance  measurement  system*  students  frequently 
forgot  the  value  of  the  ETA  which  they  last  entered. 

Using  keyboards  for  data  input  apparently  posed  a 
somewhat  greater  difficulty  for  students,  particularly 
during  early  training  problems.  Instructors  indicated  that 
three  types  of  data  input  errors  were  common.  The  most 
frequent  was  incorrect  data  entry.  The  second  was 
incorrect  formatting  of  data  which  was  entered.  Finally, 
students  occasionally  entered  correct  data  into  the 
wrong  entry  register.  It  would  appear  desirable,  therefore, 
to  strengthen  training  in  the  use  of  keyboard  entry  devices 
prior  to  the  student's  using  the  device  to  receive  navigation 
and  communication  training. 

Learning  the  uses  of  each  functional  area  of  the 
student  stations  and  learning  requirements  and  procedures  for 
controlling  the  trainer  were  rated  by  students  and  instructors 
as  being  somehwat  more  difficult.  Again,  additional 
familiarization  training  would  appear  to  be  desirable. 

In  the  training  problems,  as  they  were  originally 
designed,  an  annunciator  panel  in  each  student  station  was 
used  to  display  cues  to  students,  interrogate  students, 
and  display  knowledge  of  results  feedback.  In  practice, 
it  was  found  that  the  annunciator  messages  were  disruptive 
to  student  tasks.  Accordingly,  the  use  of  the  annunciator 
panel  was  greatly  reduced.  Additionally,  both  instructors 
and  students  complained  about  an  auditory  tone  signal 
which  occurred  when  a  message  was  removed  from  the 
annunciator.  The  tone  provided  practically  no  meaningful 
information,  aside  from  alerting  the  student  that  he  may  have 
missed  a  message,  the  content  of  which  was  unidentified. 
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Students  made  numerous  comments  regarding  what  might  be 
termed  classic  human  factors  design  aspects  of  the  student 
stations.  More  frequently  occurring  comments  are  summarized 
below. 

Although  the  work  surface  was  larger  than  any  such 
surface  which  might  be  found  in  two-place  aircraft,  students 
felt  the  surface  was  too  small  for  the  amount  of  paper  work 
required  in  navigation,  and  especially  in  dead  reckoning 
navigation.  It  is  unlikely  that  the  size  of  the  work  surface 
interfered  meaningfully  with  learning. 

Storage  space  was  felt  to  be  less  than  adequate  for 
the  amount  of  materials  which  needed  to  be  stored,  such  as 
charts  and  publications. 

Illumination  levels  in  certain  student  stations  were 
felt  to  be  low.  This  was  compounded  by  the  fact  that  the 
gooseneck  lamps  in  the  stations  did  not  appear  to  provide 
meaningful  levels  of  additional  illumination. 

Some  students  felt  that  side  consoles  were  lower  than 
they  should  be,  making  it  somewhat  awkward  to  reach  controls 
located  on  the  panels. 

The  headset  and  communications  foot  pedal  switch 
received  considerable  comment.  The  headsets  were  found 
to  be  uncomfortable  by  many  students.  Additionally,  the 
plug  for  the  student's  headset  was  located  on  the  wrong  side 
of  the  station.  As  a  result,  the  cord  draped  across  the 
student's  lap  and  occasionally  dragged  across  the  working 
surface.  The  foot  pedal  used  to  key  the  microphone  could 
not  be  easily  reached  by  short  students.  Other  students 
experienced  difficulty  by  accidentally  activating  the  switch. 

Seats  used  in  student  stations  were  the  subject  of 
considerable  comment.  The  seats  were  found  to  be  very 
uncomfortable  after  relatively  short  periods  of  sitting. 

Since  some  TPs  required  several  hours  of  sitting,  the 
problem  became  pronounced.  Additionally,  forward  seat 
travel  appeared  to  be  less  than  necessary,  particularly 
in  relation  to  the  small  working  surface.  This  required 
students  to  lean  forward  more  than  should  have  been  necessary 
to  perform  tasks  such  as  plotting  and  measuring.  When  the 
seats  were  at  their  full  aft  travel,  wheels  on  the  seats  ran 
over  the  facility  carpeting.  Ruts  in  the  carpeting  were 
beginning  to  appear  after  the  trainer  had  been  in  use  less 
than  six  months.  Finally,  the  plastic  material  used  to 
cover  the  seats  caused  considerable  perspiration. 
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COMMUNICATIONS  SYSTEM 

It  was  agreed  by  both  students  and  instructors  that 
communications  training  which  could  be  accomplished  in  the 
device  was  of  value  for  learning  the  content  and  format 
of  radio  communications.  Results  obtained  in  the  training 
effectiveness  evaluation  of  the  device  .confirmed  their  view 
(see  table  6).  Both  students  and  instructors  also  felt  that 
the  effectiveness  of  the  device  for  communications  training 
might  be  enhanced  if  simulations  of  radio  communications 
were  more  realistic.  Presently,  the  simulated  communications 
channels  are  very  clear  and  of  good  fidelity.  It  was 
strongly  felt  that  the  progressive  addition  of  static,  weak 
signals,  and  background  communications  traffic  might  result 
in  even  better  transfer  of  communication  training  .  Similar 
comments  were  made  regarding  the  simulation  of  navigation 
radio  reception. 

Device  software  was  designed  to  preclude  the  capability 
for  a  student  to  reestablish  communications  to  a  station  with 
which  he  already  had  communicated.  If  a  student  made  an 
error  during  the  first  communication  with  the  station, 
he  was  unable  to  communicate  again  to  correct  the  error  if 
he  had,  in  the  meantime,  switched  to  a  different  frequency. 

This  feature  was  found  to  be  undesirable  and  unrealistic. 

To  communicate  with  instructors  or  TDOs,  students 
activated  a  foot  switch.  This  procedure  placed  their 
communication  request  in  a  queue  with  other  Bequest  backlogs. 
Communication  with  students  was  then  handled  individually 
on  a  first  come  first  serve  basis.  Both  students  and 
instructors  suggested  the  addition  of  some  fonfttof'dEeedback 
to  the  student  to  indicate  that  his  request  was  in  a  queue. 

The  importance  of  such  feedback  takes  cn  particular  meaning 
in  light  of  the  fact  that,  during  peak  communication  periods, 
students  frequently  had  to  wait  for  up  to  15  minutes  in  queue. 

Several  suggestions  were  made  for  reducing  communication 
queues.  One  was  to  simulate  ATIS  (Airport  Taped  Information 
Service) .  A  second  recommendation  was  to  designate  specific 
TDOs  to  handle  specific  communication  functions  such 
as:  clearance  delivery,  ground  control,  departure  control, 
tower  and  center.  Presently,  each  TDO  handles  all  communications 
functions.  Instructors  also  felt  that  the  quality  of 
transmissions  by  TDOs  could  be  improved  through  functional 
allocation. 

AUTOMATED  PERFORMANCE  MEASUREMENT  CAPABILITY 

Device  1D23  incorporated  an  automated  student  performance 
measurement  system.  Experience  with  the  system  showed,  however, 
that  it  fully  satisfied  neither  student  nor  instructor  needs 
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for  straightforward,  highly  usable  feedback  on  student 
performance  and  progress.  Valid,  objective  measurement  of 
performance,  however,  is  essential  for  assessing  student 
progress,  diagnosing  learning  problems,  and  determining  when 
acceptable,  criterion-based  levels  of  performance  have  been 
achieved . 

Instructors  were  asked  to  rate  the  utility  of  the 
present  measurement  system  output  for  reconstructing  flights. 
Sixty-four  percent  rated  the  value  of  the  output  as  being 
moderate  to  high  for  reconstructing  flights.  Several 
instructors  pointed  out,  however,  that  legs  and  charts 
must  be  used  in  conjunction  with  the  printout.  They  further 
pointed  out  that  a  complete  post-mission  debriefing  and 
reconstruction  could  require  approximately  10  minutes  per 
student.  Generally,  therefore,  the  measurement  system 
output  was  used  by  individual  students  as  an  aid  in 
reconstructing  their  own  flights. 

Instructors  also  were  asked  to  rate  the  utility  of 
the  measurement  system  output  for  evaluating  student 
performance.  Forty-five  percent  rated  the  output  as  having 
minimal  or  no  value,  An  additional  45  percent  rated  the  output 
as  having  only  moderate  value.  The  need  for  improvements 
to  the  measurement  system  takes  on  added  perspective  when  it  is 
considered  that  64  percent  of  instructor  NFOs  rated  as  very  de¬ 
sirable  the  use  of  the  device  to  train  students  to  the  same, 
quantifiable  levels  of  performance  prior  to  allowing  them 
to  progress  to  in-flight  dead  reckoning  navigation  training. 
Fifty  percent  rated  as  very  desirable  the  same  use  of  the 
device  prior  to  in-flight  airways  navigation  and  communication 
training. 

Several  elements  of  the  measurement  system’s  operation 
and  output  were  found  to  contribute  to  the  system' s  evaluation 
by  instructors.  Some  measures  were  of  little  practical  value. 
Other  measures  were  overly  influenced  by  estimated  time  of 
arrival  errors.  Performance  measurement  was  not  independent 
for  each  leg,  resulting  in  cascading  effects.  Some  measures 
were  not  made  if  students  failed  to  input  necessary  information 
from  student  stations.  Measurement  of  communication  errors 
by  TDOs  appeared  to  be  less  standardized  than  would  be 
ideally  desired.  Measures  of  relative  student  performance 
were  degraded  by  the  fact  that  activated  but  unused  student 
stations  were  graded  as  perfect,  having  never  taken  off 
and,  therefore,  having  never  made  errors.  Finally,  the 
output  contained  some  quantitative  indices  which  were  under¬ 
stood  by  neither  instructors  nor  students.  Documentation 
provided  with  the  device  did  not  define  these  quantitative 
indices. 
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The  automated  performance  measurement  system  was  used 
during  the  timeframe  of  the  present  study  to  determine  whether 
students  had  achieved  criterion  performance.  During  TP  12, 
students  were  required  to  achieve  a  minimum  grade  of  3.2, 
but  the  grade  was  based  solely  upon  a  single  measure, 
circular  error  distance  from  turn  points  at  ETA  expiration. 

In  November  of  1973  it  was  learned  that  plans  had  been 
made  to  use  Device  1D23  in  a  proficiency  advancement  role. 

The  plan  called  for  students  to  achieve  certain  minimum 
standards  of  performance  in  the  device  before  they  would 
be  allowed  to  advance  to  in-flight  airways  training.  It 
was  planned,  however,  to  utilize  additional  instructors  and 
evaluate  student  performance  using  Aviation  Training  Forms 
and  in-flight  performance  criteria,  rather  than  the  automated 
performance  measurement  system. 

STAFFING  LEVELS 

Determining  personnel  requirements  involves  procedures 
which,  by  nature,  are  not  overly  objective  and  quantitative. 
The  following  information  is  presented  to  provide  guidelines 
for  estimating  manning  requirements  in  similar  devices  to 
be  designed  in  future  timeframes.  The  information  is 
quite  limited  and  reflects,  perhaps  as  much  as  anything, 
a  shortage  of  instructional  personnel  at  VT-10  during  the 
timeframe  when  the  study  was  conducted.  Subsequent  to  the 
study,  the  shortage  was  relieved  to  some  extent.  It  would 
seem  reasonable,  therefore,  to  supplement  the  information 
presented  below  with  more  current  data  regarding  acceptable 
staffing  levels. 

During  the  study,  student- to— instructor  ratios 
averaged  between  12  and  16  to  one,  depending  upon  the  phase 
of  training'  being  administered  in  the  device.  Instructors 
were  asked  how  often  the  student -to -instructor  ratio  seemed 
too  large  for  an  individual  instructor  to  be  able  to 
effectively  instruct.  Results  are  shown  in  table  11,  which 
presents  the  percent  of  instructors  who  responded  to  each 
frequency  category  for  the  three  training  phases  shown.  It 
is  apparent  from  the  table  that  the  prevailing  student— 
to-instructor  ratios  seemed  frequently  or  always  too  large  to 
a  considerable  majority  of  instructors. 

As  discussed  previously  under  the  subject  of  the 
communication  system,  student  to  TDO  ratios  also  appeared 
too  high,  at  least  for  peak  communications  periods.  Student 
to  TDO  ratios  averaged  between  six  and  eight  to  one. 
Additional  TDO  capabilities  for  peak  communications  periods 
may  have  been  desirable. 
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The  number  of  maintenance  personnel  appeared  to  be 
adequate  at  the  time  the  study  was  conducted. 

TABLE  1 1 .  SUMMARY  OF  INSTRUCTOR  EVALUATION  OF 
STUDENT  TO  INSTRUCTOR  RATIOS 


Training 

Phases 

Relative  Frequency  with  which  Student-to- 
Instructor  Ratios  deemed  too  Large 

1 

Pre-T-29 

0% 

12% 

38% 

50% 

Pre-T-39 

14% 

14% 

29% 

43% 

Pre-F-9 

33% 

0% 

34% 

33% 
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APPENDIX  A 

SYNOPSIS  OF  PREPROGRAMMED 
TRAINING  PROBLEMS 
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Contents  of  Training  Problems  (TPs)  1  through  12  are 
briefly  described  below.  The  descriptions  are  based  primarily 
upon  the  content  of  the  1D23  Briefing  Guide  (Ref  2).  The 
guide  was  published  in  June  1973. 

The  descriptions  which  follow  generally  reflect  content 
of  the  TPs  during  the  timeframe  (January  through  July  1973) 
when  data  were  collected  for  the  study.  However,  minor 
adjustments  to  TPs  were  frequent.  For  example,  grading 
criteria  for  various  TPs  underwent  frequent  changes,  as  did 
numbers  and  types  of  navigation  aids  which  were  made  available 
to  students.  Also,  equipment  failures  which  were  either 
programmed  or  manually  inserted  into  the  system  varied  from 
time  to  time.  The  descriptions  which  follow,  therefore,  are 
best  interpreted  as  reflecting  the  major  thrust  of  each  TP. 

TRAINING  PROBLEMS  1  AND  2,  DEVICE  INTRODUCTION  DESCRIPTION 

TPs  1  and  2  provided  a  highly  p re  prog ra timed  indoctrination 

in  the  operation  and  capabilities  of  the  trainer.  Each 
student  was  shown  a  student  station  and  was  talked  through 
a  programmed  sequence  on  the  use  of  device  controls,  perform¬ 
ance  of  instruments  and  displays,  and  performance  tolerance 
limits. 

TRAINING  OBJECTIVE.  The  objective  of  these  problems  was  to 
familiarize  the  trainee  with  the  controls,  indicators,  and 
procedures  to  be  used  in  order  to  practice  navigation  tasks  in 
the  device. 

FLIGHT  INFORMATION.  The  following  flight  information  was 
used: 

Charts.  Primary:  Mississippi  River  Central 

Secondary:  ONC  H-24 

Route .  NAS  Pensacola,  Saufley  VOR,  Picayune  VORTAC, 

Esler  VOR,  Laurel  VOR,  Saufley  VOR,  .S  Pensacola 

Temperature  Rise.  A  temperature  rise  of  +3  degrees  was  used. 

Preparation.  Students  were  required  to  have  preflight 
completed,  including  log  and  chart,  prior  to  entering  the 
trainer  room. 

NAVIGATION  AIDS  AVAILABLE.  A  total  of  23  navigation  aids  was 
available,  including  TACAN,  ADF,  VOR,  and  VORTAC. 

GRADING  CRITERIA.  Grading  criteria  were  not  applicable  to 
TPs  1  and  2. 
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EQUIPMENT  USED.  Students  received  a  two  hour  lecture 
prior  to  TPs  1  and  2  to  familiarize  them  with  the  various 
aspects  of  the  device.  All  capabilities  of  the  device 
were  then  demonstrated. 

PROCEDURAL  NOTES.  For  these  TPs,  students  needed  only 
to  follow  instructions  provided  by  the  instructor  on  how 
to  operate  the  device  and  interpret  displayed  feedback. 

NAVIGATION  TOOLS.  The  following  navigation  tools  were 
required:  dividers,  CR  computer,  plotter,  preflighted  chart 
and  log,  spare  logs,  IFR  supplement,  low  altitude  airways 
chart,  low  altitude  approach  plates,  paper,  and  pencil. 

TRAINING  PROBLEM  3,  DR  NAVIGATION 

DESCRIPTION.  TP  3  was  "flown"  in  an  approximated  E-2A 
type  aircraft.  Aircraft  cruise  altitude  and  airspeed 
were  controlled  by  the  training  device.  One  altitude 
change  occurred  during  the  flight.  TACAN  DME  was  failed 
part  way  through  the  mission  to  force  the  student  to  use 
multiple  LOP  (line  of  position)  position  fixing. 

TP  3  was  designed  to  exercise  the  trainee  in  the 
fundamentals  of  dead  reckoning  navigation;  use  of  the  plotter, 
divider,  hand  computer,  charts;  and  basic  procedures  to 
solve  for  position  of  the  aircraft,  direction  to  destination, 
and  time  of  arrival. 

Students  were  responsible  for  the  calculation  and 
input  of  ETAs,  winds,  and  positions  for  automated 
performance  evaluation.  They  also  were  responsible  for 
directing  the  flight  over  the  prescribed  route. 

Voice  communications  with  the  ground  included  proper 
use  of  current  procedures  for  clearance  delivery,  ground 
control,  tower,  departure  control,  and  VFR  position  reports 
to  a  Flight  Ser^i^e  Station. 

TRAINING  OBJECTIVES „  Upon  completion  of  TP  3,  it  was 

fj^af  f  Jio  Studfilit  wnn  1  ^  Ko  »K1o  fr»  • 


1.  Plot  TACAN  and  multiple  LOP  fixes  on  charts. 

2.  Operate  keyboard  to  enter  fix  information. 

3.  Operate  UHF  radio  controls. 

4.  Give  VRF  position  report. 

5.  Communicate  correctly  with  ground,  tower,  departure, 
and  Flight  Service  Station  using  proper  jargon  and 
format. 
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6.  Plot  dead  reckoning  courses# 

7.  Utilize  plotter,  dividers  ond  CR  computer, 

8.  Dead  reckon  ahead  using  real  time  information. 

9.  Complete  proper  entries  in  a  DR  in-flight 
navigation  log. 

10.  Calculate  ETAs. 

11.  Correlate  instrument  reading  and  chart  plotting. 

12.  Apply  wind  and  position  information  to  make  course 
corrections . 

13.  Determine  wind  by  track/groundspeed  and  airplot 
methods. 

14.  Operate  TACAN,  VOR  and  UHF  ADP  equipment# 

15.  Recognize  navaid  station  passage  from  cockpit 
instrument  indications. 

16.  Operate  keyboard  to  enter  wind  information 
for  grading  and  navigation  updating, 

17.  Operate  keyboard  to  enter  ETAs. 

FLIGHT  INFORMATION.  Same  ai  TP  1. 

NAVIGATION  AIDS  AVAILABLE.  Same  as  TP  1. 

GRADING  CRITERIA.  The  following  criteria  were  programmed 
into  the  device's  automated  performance  evaluation  system: 


Descriptions 

Grade 

Limits 

Positions 

4 

0-2  NM 

3 

2-4  NM 

2 

4-6  NM 

1 

>  6  NM 

winds 

4 

15®  and  5 

3 

25  and  5 

2 

30°  and  12 

1 

>30  or  12 

Turn  Points 

4 

0-3  NM 

3 

3-6  NM 

2 

6-8  NM 

1 

>  8  NM 
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Each  position,  wind,  and  turn  point  was  graded  individually 
and  then  averaged  together  with  all  other  positions,  winds, 
and  turn  point  inputs  to  develop  an  overall  performance 
average  (grade)  for  the  student. 

EQUIPMENT  USED.  The  following  equipments,  simulated 
in  the  device,  were  used:  TACAN,  VOR,  UHF  ADF,  UHF 
Command  Radio,  UHF  Auxiliary  Radio,  and  IFF. 

PROCEDURAL  NOTES.  The  student  briefing  guide  for 
Device  1D23  and  instructor  briefings  prior  to  TPs  1  and 
2  stressed  several  points.  The  points  are  addressed  below. 

Turn  point  circular  error  was  determined  by  comparing 
the  actual  position  of  the  "aircraft"  with  the  actual 
position  of  the  turn  point  at  the  time  of  ETA  expiration, 
as  calculated  and  entered  by  the  student.  The  determination 
and  entry  of  accurate  ETAs,  therefore,  was  critical  to 
achieving  successful  performance.  A  specific  step-by-step 
procedure  was  recommended  for  students  to  use  each  time  they 
took  a  fix.  Students  were  encouraged  to  continually 
update  their  ETAs. 

Aircraft  flown  at  VT-1.  displayed  an  indicated  outside 
air  temperature.  However,  the  trainer  was  designed  to  give 
true  outside  air  temperature.  Students  were  advised  of 
this  difference. 


Students  also  were  instructed  in  special  techniques 
for  folding  charts  so  that  they  would  fit  on  the  relatively 
small  working  surfaces  in  the  trainee  stations. 

Because  accomplishment  of  proper  voice  communications 
was  one  of  the  major  objectives  of  training  in  Device  1D23, 
students  were  given  explicit  voice  procedures  to  be  used. 

They  also  were  admonished  not  to  expect  an  immediate  response 
when  making  a  voice  report  because  of  high  instructor  workloads 
during  departure  and  approach  phases. 

NAVIGATION  TOOLS,  Same  as  TP  1. 


•rnAjiNING  PROBLEM  4 , 


w*\  unv  awi.« 


DESCRIPTION.  TP  4  was  flown  in  an  approximated  E-2A  type 
aircraft.  Aircraft  cruise  altitude  and  airspeed  were 
controlled  by  the  training  device. 

TACAN  DME  wa3  failed  part  way  through  the  mission  to 
force  the  student  to  use  multiple  LOP  position  fixing. 

The  construction  and  use  of  an  EP  (estimated  position)  also 
was  required. 
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As  in  TP  3,  the  student  was  responsible  for  calculating 
and  inputing  ETAs,  winds,  headings,  and  positions  while  using 
basic  dead  reckoning  procedures. 

All  voice  communications  with  the  ground  were  the 
responsibility  of  the  student. 

TP  4  also  was  designed  to  prepare  students  for  the 
roles  they  would  assume  during  in-flight  dead  reckoning 
training.  Tc  do  this,  students  were  divided  into  groups. 

One  student  in  each  group  assumed  the  foie  of 
lead  navigator  while  the  remaining  students  assumed  the 
role  of  tracker.  Students  took  turns  acting  as 
lead  navigators  during  the  mission. 

TRAINING  OBJECTIVES.  Upon  completion  of  TP  4,  it  was 
intended  that  the  student  would  be  able  to: 

1.  Apply  corrections,  computations  and  procedures 
as  outlined  in  the  TP  3  objectives. 

2.  Communicate  with  all  controlling  agencies 
using  proper  voice  procedures. 

FLIGHT  INFORMATION.  The  following  flight  information 
was  used: 

Charts .  Primary:  State  of  Florida 

Secondary:  Florida  Atlantic  Coast  Area 

Route.  NAS  Pensacola,  Crestview  VORTAC,  Tallahassee 
VORTAC,  St.  Peter sburgh  VORTAC,  Jacksonville  VORTAC, 

NAS  Mayport. 

Temperature  Rise.  A  temperature  rise  of  +4  degrees  was  used. 

Preparation.  Students  were  required  to  have  preflight 
completed,  including  log  and  chart,  prior  to  entering 
the  trainer  room. 

NAVIGATION  AIDS  AVAILABLE.  A  total  of  34  navigation  aids 
was  available,  including  TACAN,  ADF,  VOR,  and  VORTAC. 

GRADING  CRITERIA.  Same  as  TP  3. 

EQUIPMENT  USED.  Same  as  TP  3. 

PROCEDURAL  NOTES.  Procedures  for  TP  3  were  reviewed. 
Estimated  position  and  advance  and  retard  construction  and 
use  were  discussed  by  instructors. 

NAVIGATION  TOOLS.  Same  as  TP  1. 
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TRAINING  PROBLEM  5,  DR  NAVIGATION 

DESCRIPTION.  TP  5  was  flown  in  an  approximated  E-2A  type 
aircraft.  Aircraft  cruise  altitude  and  airspeed  were 
controlled  by  the  training  device. 

The  problem  included  two  airplot  phases  based  on 
pilot  evasive  action  for  weather.  It  also  included  failure 
of  TACAN  capability,  forcing  the  student  to  use  multiple 
LOP  position  fixing.  Students  also  were  required  to 
plot  three  estimated  positions  using  single  LOPs,  and 
determine  position  by  using  advance  and  retard  techniques. 

As  in  TPs  3  and  4,  the  student  was  responsible  for  the 
calculation  and  input  of  ETAs,  winds,  headings  and  positions. 

TRAINING  OBJECTIVE.  Upon  completion  of  TP  5,  it  was 
intended  that  the  student  would  be  able  to: 

1.  Apply  corrections,  computations  and  procedures 
as  outlined  in  TP  3  and  4  objectives. 

2.  Operate  UHF  controls  and  make  all  voice  reports 
required  and/or  requested, 

3.  Plot  and  log  estimated  positions, 

4.  Construct  and  compute  an  airplot  wind  while 
evading  weather. 

5.  Construct  and  utilize  advance  and  retard 
LOPs  for  position  determination, 

FLIGHT  INFORMATION.  The  following  flight  information  was 
used: 

Charts.  Primary:  Missippi  River  Central 
Secondary :ONC  H-24 

Route .  NAS  Pensacola,  Saufley  VOR,  Baton  Rouge  VORTAC, 

Sabine  Pass  VORTAC,  Leeville  VORTAC,  Saufley  VOR, 
mac  Psnsctcols. 

Temperature  Rise.  A  temperature  rise  of  +6  degrees  was  used. 

Preparation.  Students  were  required  to  have  preflight  completed, 
to  Include  log,  chart,  and  DD-175,  prior  to  entering  the 
trainer  room. 

NAVIGATION  AIDS  AVAILABLE.  A  total  of  18  navigation  aids  was 
available,  including  TACAN,  ADF,  VOR,  AND  VORTAC. 
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GRADING  CRITERIA.  Same  as  TP  3. 

EQUIPMENT  USED.  Same  as  TP  3. 

PROCEDURAL  NOTES.  Procedures  for  TPs  3  and  4  were  reviewed. 
Procedures  were  reviewed  for  construction  and  use  of  an  airplot 
while  evading  weather,  and  use  of  advance  and  retard  LOPs. 
Standard  instrument  departure  procedures  were  discussed 
during  the  premission  briefing. 

NAVIGATION  TOOLS.  Same  as  TP  1. 

TRAINING  PROBLEM  6,  DR  NAVIGATION 

DESCRIPTION.  TP  6  was  designed  as  a  DR  navigation  check  ride 
to  evaluate  student  proficiency  in  all  DR  navigation  procedures. 
It  was  flown  in  an  approximated  E-2A  type  aircraft. 

Aircraft  altitude  was  controlled  by  the  student.  On  one 
leg,  the  student  was  given  a  controlled  time  of  arrival 
problem.  The  TP  also  included  a  forced  diversion  to  an 
alternate  destination,  requiring  a  change  in  flight  plan.  As 
in  all  training  problems,  the  student  was  responsible  for  all 
voice  communications,  and  corrections  and  calculations  of  ETAs, 
winds  and  positions. 

TRAINING  OBJECTIVES.  Upon  completion  of  TP  6,  it  was  intended 
that  the  student  would  be  able  to: 

1.  Apply  corrections,  computations, and  procedures 
as  outlined  in  TP  3. 

2.  Detect  and  compensate  for  equipment  malfunctions. 

3.  Compute  and  execute  a  controlled  time  of  arrival 
problem. 

FLIGHT  INFORMATION.  The  following  flight  information  was 
used : 

Charts.  Primary:  Mississippi  River  Central 

Secondary:  ONC  H-24 

Route .  NAS  Pensacola,  Saufley,  N29-37  W92-00,  Polk  VOR, 

Jackson  VORTAC,  Saufley  VOR,  NAS  Pensacola. 

NAVIGATION  AIDS  AVAILABLE.  A  total  of  27  navigation  aids  was 
available,  including  TACAN,  ADF ,  VOR,  and  VORTAC. 

GRADING  CRITERIA.  Same  as  TP  3. 

EQUIPMENT  USED.  Same  as  TP  3. 
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NAVIGATION  TOOLS.  Same  as  TP  3 . 

PROCEDURAL  NOTES.  Procedures  for  TP  5  were  reviewed. 
NAVIGATION  TOOLS.  Same  as  TP  1. 

TRAINING  PROBLEM  7,  LOW-LEVEL  AIRWAYS  NAVIGATION 

DESCRIPTION.  TP  7  was  designed  to  exercise  the  student  in 
"Victor"  airways  navigation  and  IFR  voice  communications 
requirements  under  nonradar-contact  conditions.  It  was  flown 
in  an  approximated  E-2A  type  aircraft.  Cruising  airspeed  was 
under  program  control  and  required  altitude  changes  were  under 
trainee  control. 

The  student  was  responsible  for  all  turning  point  ETAs 
and  for  directing  the  aircraft  along  the  airways  using  FLIP 
chart  inbound/outbound  radials.  The  student  was  responsible 
for  fuel  management  and  the  use  of  a  jet  log.  Also,  students 
were  responsible  for  all  IFR  voice  communications,  both 
directed  and  mandatory. 

TRAINING  OBJECTIVES.  Upon  completion  of  TP  7,  it  was  intended 
that  the  student  would  be  able  to: 

1.  Direct  the  flight  over  low  altitude  airways. 

2.  Demonstrate  airways  procedures  including  reporting 
requirements. 

3.  Give  IFR  position  reports. 

4.  Use  aircraft  systems  (VOR/TACAN)  to  compute  ETA. 

5.  Operate  navaids  to  maintain  flight  on  radials, 

6.  Operate  navaids  to  navigate  point-to-point. 

7.  Input  necessary  heading  changes  to  maintain  course 

8.  Operate  UHF  radio  correctly. 

9.  Communicate  using  proper  jargon  and  formats, 

10.  Utilize  low  altitude  FLIP  publications  (chart, 
approach  plates,  IFR  supplement), 

11.  Properly  use  a  jet  log. 

12.  Preflight  estimated  fuel  required  and  maintain  the 
status  of  in-flight  actual  fuel  consumption. 
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FLIGHT  INFORMATION.  The  following  flight  information  was 
used: 

Charts.  L-17/18  and  L-19/20. 

Route.  NAS  Pensacola,  Crestview  VORTAC,  V-189  Tallahassee, 

V-225  Key  West,  initial  approach  fix  HI-VORTAC  07. 

Temperature  Rise.  A  temperature  rise  of  +10  degrees  was  used. 

Preparation.  Students  were  required  to  have  preflight  completed, 
including  jet  log,  L-17/18,  L-19/20,  and  DD-175,  prior  to 
entering  the  trainer  room.  Basic  information  for  the  DD-175 
was  provided  to  the  students. 

NAVIGATION  AIDS  AVAILABLE.  A  total  of  33  navigation  aids  was 
available,  including  TACAN,  ADF,  VOR,  and  VORTAC. 

GRADING  CRITERIA.  Same  as  TP  3 . 

EQUIPMENT  USED.  The  full  system  capability  of  the  device  was  used. 

PROCEDURAL  NOTES.  TP  7  involved  airways  navigation  and 
communicat j  .  The  student  was  required  to  maintain  a  desired 
radial,  navigate  TACAN  point-to-point,  communicate  on  UHF ,  and 
manage  the  fuel.  The  student  had  previously  been  taught  the 
necessary  procedures.  In  the  briefing  prior  to  the  mission, 
specific  procedures  required  for  radial  tracking  and  point-to- 
point  navigation  were  reviewed. 

For  the  mission,  students  were  advised  to  plan  turns  so  as 
not  to  overshoot  turn  points.  At  a  TAS  of  320  knots  and  a 
heading  change  greater  than  45  degrees,  they  were  advised  to 
lead  the  turn  by  about  3  nautical  miles.  Because  leading  turns 
were  necessary,  students  were  cautioned  regarding  entering  the 
next  ETA  prior  to  the  expiration  of  a  current  ETA. 

Cruise  control  data  and  normal  thrust  climb  data  for  the 
E-2A  simulation  were  provided  to  the  students. 

NAVIGATION  TOOLS.  The  following  navigation  tools  were 
required:  CR  computer,  completed  jet  logs,  low  altitude 
airways  charts,  DD-175,  low  altitude  approach  plates,  high 
altitude  approach  plates,  enroute  supplement,  pencil  and  paper. 

TRAINING  PROBLEM  8,  HIGH  ALTITUDE  AIRWAYS  NAVIGATION 

DESCRIPTION.  TP  8  was  flown  in  an  approximated  F-4J  aircraft. 
Cruising  airspeed  was  under  program  control;  required  altitude 
changes  were  under  student  control. 
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TP  8  was  designed  to  exercise  the  student  in  high  altitude 
airways  navigation  in  a  radar  contact  environment.  The  TP 
reflected  a  continuing  emphasis  on  IFR  requirements  and 
procedures.  Students  were  required  to  initiate  an  in-flight 
refiling  of  clearance.  This  resulted  in  the  need  to  change 
preflighted  data.  Students  were  responsible  for  flight 
direction  along  airways  as  initially  cleared  and  to  the 
intersection  of  a  new  airway  after  refiling  clearance. 

Students  also  were  responsible  for  all  IFR  voice  communications, 
both  directed  and  mandatory <, 

TRAINING  OBJECTIVES.  Upon  completion  of  TP  8,  it  was  intended 
that  students  would  be  able  to: 

1*  Control  information  sources  and  operate  equipment  as 
outlined  in  TP  7. 

2.  Direct  the  flight  over  high  altitude  airways  (radial 
tracking)  and  navigate  using  navaid  offsets  from  the 
airways  (point-to-point  navigation), 

3.  Correctly  apply  airways  procedures  and  voice 
communications  procedures. 

4.  Use  high  altitude  FLIP  publications  (charts,  approach 
plates,  IFR  supplement). 

FLIGHT  INFORMATION,  The  following  flight  information  was 
used: 

Charts.  H-3/4 . 

Route.  NAS  Pensacola,  Crestview  VORTAC,  J-50  HcComb  VORTAC, 
Jackson  VORTAC,  Greenwood  VORTAC,  Memphis  VORTAC,  J-39  Crest- 
view  VORTAC,  initial  approach  fix  NPA  (runway  06)  NAS  Pensacola. 

Temperature  Rise.  A  temperature  rise  of  +25  degrees  was  used. 

Preparation.  Students  were  required  to  have  preflight 
completed ,  including  jet  log,  H-3/4  and  DD-175,  prior  to 
entering  the  trainer  room.  Basic  information  for  the  DD-175 
was  provided  to  the  students. 

NAVIGATION  AIDS  AVAILABLE.  A  total  of  37  navigation  aids  was 
available,  including  TACAN,  ADF,  VOR,  and  VORTAC. 

GRADING  CRITERIA.  Same  as  TP  3. 

EQUIPMENT  UTILIZED.  The  full  system  capability  of  the  device 
was  used. 
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PROCEDURAL  NOTES.  Procedural  notes  were  the  same  as  those  for 
TP  7,  except  that  students  were  advised  to  lead  turns  by  4 
nautical  miles  when  a  heading  change  greater  than  45  degrees 
was  required. 

NAVIGATION  TOOLS.  The  following  navigation  tools  were 
required:  CR  computer,  completed  jet  logs,  high  altitude 
airways  charts,  DD-175,  high  altitude  approach  plates,  enroute 
supplement,  and  paper  and  pencil. 

TRAINING  PROBLEM  9,  OPERATIONAL  NAVIGATION  MISSION 

DESCRIPTION.  TP  9  was  a  simulated  bombing  mission  in  an 
approximated  A-6E  aircraft.  It  was  designed  to  exercise  the 
student  in  the  use  of  advanced  navigation  systems  found  in 
modern  aircraft. 

TP  9  involved  a  launch  from  an  aircraft  carrier,  followed 
by  climb  to  altitude  and  departure  point  using  the  carrier 
TACAN.  A  target  time  was  given  to  the  student  prior  to 
departing  on  the  enroute  phase.  The  student  exercised  control 
over  aircraft  speed  to  make  the  target  time  good. 

All  navigation  systems  were  operational,  but  radio  navaids 
were  limited  to  the  carrier  TACAN  and  one  land  based  alternate. 

Upon  completion  of  the  target  run,  the  student  directed 
the  flight  back  to  the  carrier  utilizing  available  navigation 
systems.  The  student  was  given  a  marshal  point  after 
establishing  voice  communications  with  the  carrier  at  40 
nautical  miles. 

The  student  was  responsible  for  enroute  navigation,  fuel 
control,  airspeed  and  altitude  control,  and  proper  utilization 
and  validation  of  navigation  systems. 

TRAINING  OBJECTIVES.  Upon  completion  of  TP  9,  it  was  intended 
that  students  would  be  able  to  execute  the  following: 

1.  Utilize  navigation  systems  to  check  position, 
maintain  course,  and  meet  turn  point  and  target  ETAs, 

2.  Make  appropriate  voice  reports  to  controlling  agencies. 

3.  Manage  fuel  and  not  exceed  required  minimuras, 

4.  Adjust  airspeed  to  make  good  ETAs. 

FLIGHT  INFORMATION.  The  following  flight  information  was  used: 
Chart.  ONC  H-25 . 
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Route.  Ship  (29-OON,  76-OOto) ,  rendezvous  point  (29-07N, 

76-27W) ,  coast  in  point  (30-28N,  81-25W) ,  initial  point 
(28-53N,  82-36W) ,  target  (27-53N,  82-34W) ,  coast  out  point 
(27-14N,  80-12W) ,  A/H  to  marshall  (28-32N,  76-32W) ,  marshal 
(carrier  TACAN  radial  and  DME) ,  Ship. 

Temperature  Rise.  The  navigation  computer  was  used  to  compute 
TAS. 

Preparation.  Students  were  required  to  have  preflight 
completed ,  including  attack  log  and  ONC  chart,  prior  to 
entering  the  trainer  room. 

NAVIGATION  AIDS  AVAILABLE.  The  Glynco  TACAN  and  the  8hip  TACAN 
were  available. 

GRADING  CRITERIA.  Same  as  TP  3 ,  with  the  addition  of  a  bombing 
grade,  which  is  defined  below. 

Grade  Limits 

Bullseye 
<1  NM 
1-2  NM 
>2  NM 

EQUIPMENT  UTILIZED.  The  full  system  capability  of  the  device 
was  used. 

PROCEDURAL  NOTES.  Thin  was  the  first  training  problem  that 
required  students  to  utilize  aircraft  navigation  systems 
almost  exclusively.  Only  one  TACAN  station  was  used  to  update 
the  navigation  computer.  Specific  procedures  for  utilizing 
the  station  were  given  to  the  students. 

NAVIGATION  TOOLS.  The  following  navigation  tools  were 
required:  CR  computer,  completed  attack  log,  ONC  chart, 
paper  and  pencil. 

TRAINING  PROBLEM  10/11,  CHECK  MISSION 

DESCRIPTION.  TP  10/11  was  flown  in  an  approximated  F-4J 
aircraft  with  all  navigation  systems  operational.  It  was  a 
check  ride  to  evaluate  student  proficiency  in  all  areas  of 
navigation  taught  at  VT-10. 

TP  10/11  consisted  of  two  independent  problem  phases. 

Phase  one  (TP  10)  was  an  IFR  airways  flight  and  was  similar  to 
TP  8  with  respect  to  airways  navigation,  voice  communications, 
and  airways  aircraft  control.  Phase  two  (TP  11)  involved  a 
departure  from  Patrick  AFB  to  intercept  an  aircraft  carrier. 

It  was  similar  to  the  latter  part  of  TP  9. 
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The  two  phases  were  separated  so  that  all  students 
concluded  the  first  phase  before  any  students  started  the 
second  phase.  The  interim  time  between  phases  was  scheduled 
for  flight  planning  and  preflighting  of  the  second  phase. 

TRAINING  OBJECTIVES.  Upon  completion  of  TP  10/&1,  it  was 
intended  that  the  student  would  be  able  to: 

1.  Operate  all  navigation  and  communication  equipment 
as  outlined  in  TPs  3  through  9. 

2.  Utilize  all  navigation  systems  to  check  positions, 
maintain  course,  and  meet  ETAs. 

3.  Make  appropriate  voice  reports  to  all  controlling 
agencies . 

4.  Correctly  complete  appropriate  logs  and  forms. 

5.  Manage  fuel  and  understand  rainimums  required. 

FLIGHT  INFORMATION.  The  following  flight  information  was 
used: 

Charts.  TP  10:  K-3/4. 

TP  11:  ONC  H-25, 

Routes.  TP  10:  NAS  Pensacola,  Crestview  VORTAC,  J-2 
Tallahassee  VORTAC,  J-20  Orlando  VORTAC,  Patric  AFB  initial 
approach  fix  (Polaris) . 

TP  11;  Patrick  AFB,  Sturgeon  intersection  (28-34N, 
78-51W) ,  A/H  to  Ship  (24-46N,  76-26W)  . 

Temperature  Rise.  The  navigation  computer  was  used  to  compute 

T_t 

Preparation.  Students  were  required  to  have  preflight 
completed” including  jet  log,  chart,  and  DD-175,  prior  to 
entering  the  trainer  room  for  either  TP. 

NAVIGATION  AIDS  AVAILABLE.  Ten  navigation  aids  were  available, 
including  TACAN,  ADF ,  VOR  and  VORTAC. 

GRADING  CRITERIA.  Same  as  TP  3 . 

EQUIPMENT  UTILIZED.  The  full  system  capability  of  the  device 
was  used. 

PROCEDURAL  NOTES.  The  DD-175  did  not  reflect  the  route  planned 
because  the  ship's  position  was  not  given  prior  to  TP  11.  The 
DD-175  for  TP  ii  was  discussed  during  the  premiss ion  briefing. 
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NAVIGATION  TOOLS.  Same  as  TPs  8  and  9. 

TRAINING  PROBLEM  12,  POINT-TO-POINT  NAVIGATION 

DESCRIPTION.  TP  12  was  designed  to  instruct  and  test  the 
student  in  point-to-point  navigation.  The  student  had  full 
control  of  airspeed,  altitude  and  heading.  A  total  of  eight 
points  was  used.  The  student  was  required  to  score  3.2  or 
higher  (average  for  all  points)  prior  to  advancing  to  in-flight 
training  in  the  T-39  aircraft.  Each  student  was  given  three 
chances  to  qualify  prior  to  his  being  interviewed  by  the 
Navigation  Branch  Officer. 

TRAINING  OBJECTIVES.  Upon  completion  of  TP  12,  it  was  intended 
that  the  student  would  be  able  to: 

1.  Operate  navaids  to  navigate  point-to-point 
proficiently  (score  3.2  or  higher), 

2.  Calculate  and  input  ETAs  for  grading  purposes. 

3.  Calculate  groundspeed  and  wind  mentally. 

FLIGHT  INFORMATION.  The  following  flight  information  was  used: 
Charts.  No  charts  were  needed. 

Route.  Annunciator  messages  were  used  to  describe  the  radial 
and  DME  for  each  point. 

Temperature  Rise.  A  temperature  rise  of  +14  degrees  was  used. 

Preparation.  No  preparation  of  charts  or  forms  was  required. 

NAVIGATION  AIDS  AVAILABLE.  A  total  of  40  navigation  aids  was 
available,  including  TACAN,  ADF,  VOR,  and  VORTAC. 

GRADING  CRITERIA.  The  following  grading  system  was  applied  for 
distance  from  a  point  at  the  expiration  of  ETA: 

Grade  Limit 


4  <2  NM 

3  2.1  NM  -  4.0  NM 

2  4.1  NM  -  6.0  NM 

1  >6.0  NM 

PROCEDURAL  NOTES.  Students  were  required  to  determine  winds 

mentally.  Students  were  required  to  calculate  an  indicated 
airspeed  which  would  give  a  true  airspeed  of  360  knots  at 
16,000  feet. 
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Students  were  advised  that  ETAs  were  critical  to  the 
computation  of  their  grade.  Keesler,  Hattie sburg/  and  Navy 
Meridian  TACANs  were  utilized.  The  student  was  given  control 
of  the  flight  at  500  feet  of  altitude. 

NAVIGATION  TOOLS.  The  following  navigation  tools  were 
required:  OR  computer  (used  only  to  calculate  IAS) ,  paper 
and  pencil. 


76 


NAVTRAEQUIPCEN  72-C-0209-2 


APPENDIX  B 
CONTENT  OP 

INSTRUCTOR  QUESTIONNAIRE 
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Name 


Rank 


Please  check  one:  NFO  Pilot 


Indicate  the  month  and  year  when  you  began  instructing  at 
VT-10 : 


Check  the  system (s)  in  which  you  presently  are  qualified 
to  instruct.  T-29  T-39  1D23 


Over  the  past  two  months  only,  estimate  the  percentages 
of  your  training  hops  you  flew  in  each  of  the  following 
aircraft: 


T-29  %  T-39  % 


At  the  time  when  training  in  Device  1D23  was  initiated 
(December  1972) ,  estimate  the  percentages  of  your 
training  hops  you  flew  in  each  of  the  following  aircraft: 

T-29  %  T-39  % 


How  many  student  classes  have  you  instructed  in  Device 
1D23? 


Questionnaire  and  rating  scale  items  which  follow  have  been 
organized  into  three  main  sections.  Please  read  the  brief 
statement  at  the  beginning  of  each  section.  You  need  complete 
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SECTION  1.  IMPACTS  OF  TRAINING  IN  DEVICE  1D23  UPON  STUDENT 
PERFORMANCE  IN  THE  T-29  FLIGHT  STAGE. 

You  should  complete  this  section  only  if  you  were  instructing 
in  T-29  aircraft  during  the  period  (December  1972-January  1973) , 
when  training  in  Device  1D23  was  initiated. 

1.1  Overall,  how  much  value  do  you  feel  training  in  Device 
1D23  has  been  in  preparing  students  for  T-29  flights? 

(Place  an  "x"  squarely  in  one  category) . 


Negative  No  Minimal  Moderate  High 

value  Value  Value  Value  Value 

1.1  The  following  item  is  intended  to  examine  the 
approximate  magnitude  of  any  changes  in  student  performance 
which  may  have  been  evident  following  the  introduction  of 
training  in  Device  1L23.  The  four  current  B-series  flights 
are  listed  below.  In  the  space  to  the  right  of  each  flight, 
please  write  the  B-series  flight  on  which  comparable 
student  performance  was  typical  before  the  introduction  of 
Device  1D23.  For  example,  if  you  feel  that  it  used  to  take 
through  the  B-3  flight  before  students  were  performing  s 
well  as  they  now  are  on  their  B-l  flight,  then  you  would 
write  "B-3"  in  the  space  to  the  right  of  the  B-l  flight. 

If  on  the  ether  hand,  you  feel  that  performance  on  present 
B-l  flights  is  virtually  the  same  now  as  it  was  before  Device 
1D23  was  introduced,  then  you  would  write  "B-l"  in  the  space 
to  the  right  of  the  B-l  flight. 

Presently  Observed  Performance  Flight  on  Which  Comparable 
Levels  Performance  Levels  Were 

Typical  Before  Device  1D23 

B-l 

B-i 

B-3 

B-4 


1.3  Using  the  rating  scale  on  the  following  page,  place  "X's" 
squarely  within  the  ratings  which  best  reflect  your  feelings 
about  how  well  training  in  Device  1D23  prepares  students  for 
T-29  training  flights. 

Several  task  categories  are  presented.  For  each  task 
category,  please  indicate  your  feelings  about  how  well  training 
in  che  simulator  prepares  students  to  execute  the  task  category 
during  T-29  training  flights. 
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Your  second  task  is  to  continue  by  making  separate  ratings 
for  each  task  category. 


Simply  repeat  the  above  procedure  until  you  have  completed 
all  of  the  ratings.  Then  complete  the  questionnaire  items 
on  the  following  pages. 


PREFLIGHT  PLANNING  S  PREPARATION 

Interpreting  charts . 

Identifying  restricted  areas  . 

Plotting  courses  . 

Using  correct  symbology  .  .  . 
Selecting  radio  nav  aids  .  .  . 
Preflighting  logs.  ...... 

Preparing  flight  plans  .  .  .  . 
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MEASURING  AND  COMPUTING 

Using  CR  2/3  computer  . 

Using  plotter  &  divider . 

Determining  TAS  . 

Determining  ground  speed  . 

Determining  ETA  . 

Determining  wind  direction/velocity 
Determining  drift  angle . 


NAVIGATING 

Dead  reckoning  procedures.  .  . . 

Understanding  radials  from  nav  aids . 

Plotting  accurate  TACAN  fixes . 

Plotting  multiple  LOPs  . 

Advancing/retarding  LOPs  . 

Plotting  EPs  . 

Applying  variation  ...  . 

Converting  between  Mag  &  true  heading.  .  .  . 
Plotting  track,  no-wind,  wind  lines  t  .  .  . 

Using  correct  DR  symbology  . 

Dead  reckoning  ahead  . 

Correctly  applying  drift  angle  . 

Determining  headings  to  fly.  ...  . 

Filling  out  logs  . 

Communicating  with  pilot  . 
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INTEGRATION  OF  KNOWLEDGE 

Getting  it  all  together  . 

Interpreting  instruction . 

Keeping  oriented  relative  to  aircraft  . 

position 

Keeping  oriented  relative  to  direction . 

to  checkpoints 

Pacings  tasks  to  "keep  ahead  of  the  . 

aircraft" 

Anticipating  what  would  oocur  in . 

flight 

Understanding  spatial  relationships  . 

Identifying  incorrect  nav.  inputs  or . 

solutions 

Identifying  procedural  and  computationsl.  .  .  . 
errors 
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lv  4  Do  you  feel  that  additional  training  in  Device  1D23 
could  be  substituted  for  any  of  the  B-series  fliqhts? 

1.5  If  your  answer  to  1.4  was  yes,  indicate  which  flight (s) 
you  feel  could  be  dropped  in  lieu  of  additional  simulator 
training. 

1.6  If  your  answer  to  1.4  was  yes,  what  type  is)  of  additional 
simulator  training  do  you  feel  could  be  substituted  for  the 
B-series  flights  indicated  in  your  response  to  1.5? 

1.7  If  your  answer  to  1.4  was  no,  indicate  the  factors 
associated  with  T-29  in-flight  training  which  '-annot  be 
compensated  for  by  additional  simulator  trainina. 

1.8  If  your  answer  to  1.4  was  no,  suggest  modifications  to 
the  simulator  or  the  TPs  which  might  make  it  possible  to 
substitute  simulator  training  for  in-flight  training. 

1.9  What  elements  of  present  simulator  training  do  you  feel 
have  the  best  value  in  preparing  students  for  T-29  training 
flights? 

1.10  What  elements  of  present  simulator  trainina  do  you 
feel  have  the  least  value  in  preparing  students  for  T-29 
training  flights? 
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1.11  Using  the  rating  scale  below,  rate  the  concept  of 
usinq  Device  1D23  to  train  all  students  to  at  least  the 
same,  standardized  performance  levels  prior  to  allowing 
them  to  advance  to  T-29  in-flight  training. 


Very  Possibly  No  Possibly  Very 
Undesirable  Undesirable  Opinion  Desirable  Desirable 

1.12  Please  feel  free  to  make  additional  comments  about 
training  in  Device  1D23  in  preparation  for  T-29  in-flight 
training. 


SECTION  2.  IMPACTS  OF  TRAINING  IN  DEVICE  1D23  UPON  STUDENT 
PERFORMANCE  IN  THE  T-39  FLIGHT  STAGE. 

You  should  complete  this  section  only  if  you  were 
instructing  in  T-39  aircraft  during  the  period  (December  1972- 
January  1373),  when  training  in  Device  1D23  was  initiated. 

2.1  Overall,  how  much  value  do  you  feel  training  in  Device 
1D23  has  been  in  preparing  students  for  each  T-39  training 
flight?  Please  make  a  rating  for  each  flight.  (Place 
an  "x"  squarely  in  categories) . 


Value  Value  Value  Value  Value 

2.2  Using  the  rating  scale  on  the  following  page,  place  "Xs" 
squarely  within  the  ratings  which  best  reflect  your 
feelings  about  how  well  training  in  Device  1D23  prepares  stu¬ 
dents  for  T-39  training  flights. 

Several  task  categories  are  presented.  For  each  task 
category,  please  indicate  your  feelings  about  how  well  train¬ 
ing  in  the  simulator  prepares  students  to  execute  the  task 
category  during  T-39  training  flights. 

Your  second  task  is  to  continue  by  makinq  separate  ratings 
for  each  task  category. 

Simply  repeat  the  above  procedure  until  you  have  completed 
ell  of  the  ratings.  Then  complete  the  questionnaire  items 
on  the  following  pages. 
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PREFLIGHT  PLANNING  &  PREPARATION 

Interpreting  airways  charts  . 

Selecting  radio  nav  aids 
Preflighting  logs 
Preparing  flight  plans #  ] 

MEASURING  AND  COMPUTING 

Determining  TAS  .  ,  .  . 

Determining  ground  speecl . .  . 

Determining  ETA  . 

Determining  wind  direction  &  velocity 
Determining  drift  angle  . 

NAVIGATING 

Understanding  radials  from  nav  aids  . 

Interpreting  RMI . 

Plotting  accurate  TACAN  fixes  ,  .  .  , 
Correctly  applying  drift  angle.  .  .  . 
Determining  headings  to  fly  .  .  .  .  . 

Departure  procedures . 

Enroute  procedures  . 

Turn  point  procedures  . 

Directing  flight  in  holding  pattern  . 

Approach  procedures  . 

Fuel  management  . 

Filling  out  logs  . 

Communicating  with  the  pilot  .  .  .  . 

COPILOT  DUTIES 

Interpreting  flight  instruments  .  .  . 
Interpreting  engine  instruments  .  .  . 

Performing  checklist  items  . 

Tuning  radios  . 

Setting  IFF  codes  . 

Monitoring  UHF  radios  . 

Communicating  with  the  ground  .  .  .  . 
Copying  clearances . 

INTEGRATION  OF  KNOWLEDGE 

Getting  it  all  together  . 

Interpreting  instruction . 

Keeping  oriented  relative  to  .  .  .  . 
aircraft  position 

Keeping  oriented  relative  to . 

direction  to  checkpoints 
Pacing  tasks  to  "keep  ahead  of  the.  . 
aircraft" 
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INTEGRATION  OF  KNOWLEDGE  (continued) 

Anticipating  what  occurs  in  flight . 

Understanding  spatial  relationships  .  .  .  .  . 

Identifying  incorrect  Nav  inputs/.  . 

solutions 

Identifying  procedural  &  computation . 

errors 
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2.3  Do  you  feel  that  additional  training  in  Device  1D23 
could  be  substituted  for  any  of  the  present  C-series  flights? 

2.4  If  your  answer  to  2.3  was  yes,  indicate  which 

flight (s)  you  feel  could  be  dropped  in  lieu  of  additional 
simulator  training _ . 

2.5  If  your  answer  to  2.3  was  yes,  what  type(s)  of 
additional  simulator  training  do  you  feel  could  be 
substituted  for  the  C-series  flights  indicated  in  your 
response  to  2.4? 

2.6  If  your  answer  to  2.3  was  no,  indicate  the  factors 
associated  with  T-39  fn-f Tight  training  which  cannot  be 
compensated  for  by  additional  simulator  training. 

2.7  If  your  answer  to  2.3  was  no,  suggest  modifications 

to  the  simulator  or  the  TP3  which  might  make  it  possible  to 
substitute  simulator  training  for  in-flight  training. 

2.8  What  elements  of  present  simulator  training  do  you 
feel  have  the  best  value  in  preparing  students  for  T-39 
training  flights? 

2.9  What  elements  of  oresent  simulator  training  do  you 
feel  have  the  least  value  in  preparing  students  for  T-39 
training  flights? 

2.10  Using  the  rating  scale  below,  rate  the  concept  of  using 
Device  1D23  to  train  all  students  to  at  least  the  same, 
standardized  performance  levels  prior  to  allowing  them  to 
advance  to  T-39  in-flight  training. 


Very  Possibly  No  Possibly  Very 

Undesirable  Undesirable  Opinion  Desirable  Desirable 
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2.11  Please  feel  free  to  make  additional  comments  about 
training  in  Device  1D23  in  preparation  for  T-39  in-flight 
training . 


SECTION  3.  DESIGN  FEATURES  OF  DEVICE  1D23 

You  should  complete  this  section  only  if  you  are  qualified 
to  instruct  in  Device  1D23. 

3.1  When  you  instruct  in  Device  1D23,  on  the  average,  how 
many  students  at  a  time  do  you  instruct  during  each  of  the 
following  phases: 


Pre-T-29 

Pre-T-39 

Pre-F-9 


3.2  flow  often  do  the  above  numbers  of  students  seem  to  large 
for  an  individual  to  effectively  instruct?  Please  respond 
for  each  ohase  shown  below. 


3.3  In  the  above  item,  if  you  checked  any  response  other 
than  "never",  olease  indicate  the  circumstances  (TPs,  problem 
areas,  etc.)  which  make  the  student-to-instructor  ratio  seem 
too  high. 

Pre-T-29  Phase: 

Pre-T-39  Phase: 

Pre-F-9  Phase: 

3.4  What  problems,  if  any,  have  you  experienced  in  using  the 
communication  link  (ICS)  between  students  and  instructors? 

3.5  Using  the  following  rating  scale,  place  an  "x"  in  the 
category  which  best  reflects  your  experience  regarding 
tho  value  of  the  computer  printout  of  student  performance 
for  reconstructing  the  flight. 


No  Minimal  Moderate  High 

Value  Value  Value  Value 
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3.6  Using  the  following  rating  scale,  place  an  "x"  in  the 
cateqory  which  best  reflects  your  experience  regarding  the 
value  of  the  computer  printout  of  student  performance  for 
evaluating  student  proficiency. 


No  Minimal  Moderate  High 

Value  Value  Value  Value 

3.7  From  an  instructor's  standpoint,  what  are  three  of  the 
most  important  improvements  which  could  be  made  to  the 
design  of  Device  1D23? 

3.8  From  an  instructor’s  standpoint,  what  are  three  of 
the  most  desirable  design  features  which  are  incorporated 
in  Device  1D23? 

3.9  This  item  addresses  the  period  when  students  are  first 
introduced  to  Device  1D23  and  begin  to  receive  training 

in  the  device  (e.g.  TPs  1-4) .  Using  the  rating  scale  below, 
place  "Xs"  in  the  appropriate  columns  to  indicate  your 
assessment  of  the  degrees  of  difficulty  which  students 
experience  in  each  of  the  areas  listed. 


Levels  of  Difficulty 


3.10  Please  describe  any  other  areas  where  students  appear 
to  have  difficulty  when  they  first  begin  to  use  Device  1D23. 

3.11  Please  feel  free  to  make  any  additional  comments  about 
the  design  of  Device  1D23. 
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MULTIVARIATE  STATISTICS 

Experiments  involving  complex  human  performance  often 
incorporate  a  battery  of  performance  measures  in  an  attempt 
to  reasonably  describe  the  variety  of  complex  human  behaviors 
involved.  As  a  result,  the  researcher  may  be  faced  with  a 
difficult  data  analysis  task  and  often  may  experience  a 
considerable  challenge  in  integrating  voluminous  information 
into  meaningful  forms. 

A  multivariate  data  analysis  technique  was  used  in  this 
study  because  it  provided  a  means  of  consolidating  experimental 
data,  composed  of  a  variety  of  measures,  into  a  single  analysis. 
The  multivariate  analysis  provided  a  statistical  test  which  was 
based  on  a  composite  measure.  The  composite  measure  was  based 
on  a  linear  combination.  Weightings  in  the  linear  combination 
had  been  designed  to  produce  a  composite  measure  which  would 
optimally  discriminate  between  experimental  groups. 

The  alternative  approach,  the  univariate  method,  requires 
analysis  of  each  measure  separately.  The  researcher  must  then 
combine  results  of  the  separate  analyses. 

Computer  analysis  methods  employed  in  this  study  were 
designed  to  give  both  multivariate  and  univariate  test  results. 
Accordingly,  the  following  information  was  available: 

A  statistical  test  (F-test)  for  composite 
measurement  (measurement  which  combined 
the  original  measures  to  provide  optimum 
discriminability  between  experimental 
groups) 

A  separate  statistical  test  (F-test)  for 
each  measure 

FACTORIAL  DISCRIMINATION  ANALYSIS 

The  specific  statistical  analysis  technique  used  was 
based  on  a  method  called  Factorial  Discriminant  Analysis 
(FACDIS)  by  Cooley  and  Lohnes  (Ref  5) .  The  technique 
combined  a  multiple  discriminant  analysis  (best  linear 
functions  of  measures  for  describing  group  differences)  with  a 
two-way  factorial  analysis  of  variance.  In  the  multiple 
discriminant  analysis,  the  loadings  of  each  measure  on  each 
discriminant  function  were  output.  This  was  done  so  that,  if 
groups  were  found  to  be  significantly  different,  the  differences 
could  be  interpreted  in  terms  of  the  relative  loadings  of 
measures  which  contributed  most  to  discriminating  between 
groups . 
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ANALOGY  TO  UNIVARIATE  STATISTICAL  METHODS.  The  univariate 
analysis  of  variance  involves  the  computation  of  sum  of 
squares  for  each  experimental  comparison  and  for  an  experimental 
error;  similar  computations  compose  the  multivariate  analysis 
of  variance  except,  instead  of  sums  of  squares,  sums  of  squares 
and  cross  products  (SSCP)  matrices  are  computed.  The  sms 
of  squares  for  each  measure  appears  on  the  diagonal,  while 
the  suras  of  cross  products  between  all  possible  pairs  of 
measures  are  off  the  diagonal.  Consequently,  the  mechanics 
of  the  two  methods  are  similar ,  one  involving  suras  of  squares , 
the  other  SSCP  matrices. 

In  the  univariate  method,  suras  of  squares  a.  e  divided 
by  number  of  degree.,  of  freedom  to  produce  mean  squares 
(MS) ,  and,  an  F-ratio  is  formed  by  dividing  the  MS  for  an 
experimental  treatment  by  the  MS  for  experimental  error. 

For  the  multivariate  approach  (one-way) ,  the  determinant  for 
the  within  -  SSCP  (|w|)  is  divided  by  the  determinant  for  the 
total  -  SSCP  ( |T{ ) .  Determinants  of  matrices  are  used  instead 
of  mean  square  computations;  otherwise  the  approaches  are 
similar.  The  result  of  the  ratio  of  determinants  has  been 
termed  WILKS*  LAMBDA  after  the  inventor  of  this  statistic. 

There  is,  however,  a  transformation  of  the  Wilks'  Lambda  stat¬ 
istic  to  produce  an  approximation  of  the  conventional  F 
statistic.  Thus,  one  can  test  statistical  significance  using 
the  common  tables  for  the  F  distribution. 

ONE-WAY  MULTIVARIATE  TEST  (MANOVA) .  Wnile  a  factorial  analysis 
was  conducted,  a  one-way  multivariate  analysis  of  variance  pro¬ 
gram  (MANOVA)  was  used  to  compute  basis  matrices  needed  for 
the  factorial  discriminant  analysis  program  (FACDIS) .  The 
total  SSCP  matrix  (deviations  of  all  subjects  from  the  grand 
centroid)  was  computed;  this  variance  was  then  partitioned 
into  "among-groups"  and  "within-groups"  parts; 

T  (total)  =  A  (amoiig-groups)  +  W  (within-groups) .  The 
T  and  W  matrices,  along  with  a  complete  set  of  centroids,  are 
used  as  input  for  the  FACDIS  program. 


The  MANOVA  program  also  computes  an  extension  of  Bartlett's 
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TWO-WAY  MULTIVARIATE  ANALYSIS  OF  VARIANCE  (FACDIS).  The 
factorial  model  used  by  the  FACDIS  program  is 

T  =  Ar  +  Ac  +  Ai  +  W 


where  T.  Ar,  Ac,  Ai  and  W  are  the  total,  among-rows,  among- 
columns,  interaction  and  within-groups  sums  of  squares  and 
cross-rroducts  matrices.  The  test  criteria  were; 
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hypothesized  row  effect  Ar=  |wj/|Ar  +  W| 

hypothesized  column  effect  A  c  -  |w|/[Ac  +  W| 

hypothesized  interaction  effect  A  i  =  | W | / | Ai  +  W| 

Vertical  bars  in  the  equations  indicate  determinants  of 
the  enclosed  matrices.  For  each  A  (Wilks'  Lambda),  an 
approximate  F  -  ratio  was  computed,  along  with  degrees  of 
freedom,  to  allow  table  lookup  in  the  same  way  as  for 
univariate  analysis  of  variance. 
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APPENDIX  D 

QUESTIONNAIRES 
COMPLETED  AFTER  FLIGHTS 


4 
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STUDENT  POST-FLIGHT 
QUESTIONNAIRE 


Student 1 s  Name 

Were  you  lead  Nav  Circle  this  Flight 

this  flight?  Yes  No  B1  B2  B3  B4  Cl  C2  C3 

Your  responses  to  the  following  items  will  in  no  way  influence 
your  flight  grades.  Please  respond  to  all  ituns  frankly  and 
openly. 


1.  Indicate  your  overall  estimate  of  yo< r  performance 
on  this  flight.  Place  an  "X"  square. .y  in  one 
category. 


Unac-  Marginally  Below  Average  Above  Very  Outstand 
ceptable  Acceptable  Average  Average  Good  ing 

2.  Indicate  the  degree  of  confidence  which  you  felt 
in  performing  the  following  tasks  during  this 
flight: 


♦Communicating  with  ground.  . 
Using  the  computer  .  .  .  .  . 

Plotting  fixes  .  .  . 

D  R  ahead  . 

Determining  winds . 

Giving  pilot  headings.  .  .  . 

Determining  ETAs  . 

Making  log  entries  . 


3.  Indicate  your  overall  estimate  of  how  well  vou  will 
do  on  your  next  training  flight.  Place  an  "X" 
squarely  within  one  category. 


None  Low  Moderate  High 


- 1 

Unac-  Marginally  Below  Average  Above  Very  Outstand- 

ceptable  Acceptable  Average  Average  Good  ing 
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STUDENT  CRITIQUE  FORM 
(Completed  by  Instrustcrs) 

Student  Name  _  INFO/IP  Name  _ 

Was  studen  lead  Nav  Circle  this  Flight 

this  flight?  Yes  No  Bi  B2  B3  B4  Cl  C2  C3 

1..  Was  the  student  ever  disoriented  to  the  point  that  he 
appeared  unsure  of  his  present  position?  Yes  No 

2.  How  many  times  did  the  student  ask  for  help?  _ 

3.  How  many  times  did  you  have  to  intercede  to  help  the  student 

overcome  navigation  talk  errors?  _ 

4.  How  many  times  was  the  student  unable  to  interpret  your 

instructions?  _ 

5.  Indicate  your  overall  estimate  of  the  student's  perform¬ 
ance  relative  to  the  performance  level  you  generally 
observe  for  this  flight.  Place  an  "X"  squarely  within  one 
category. 


Unae-  Marginally  Below  Average  Above  Very  Cutstand- 

ceptable  Acceptable  Average  Average  Good  ing 


6.  Please  estimate  the  amount  of  hesitation  the  student 
exhibited  in  performing  the  following  tasks. 

None  Low  Moderate 

^Communicating  with  ground.  .  . 

Using  the  computer  . 

Plotting  fixes  ........ 

D  R  ahead . .  . 

Making  log  entries  ...... 


7.  Indicate  your  overall  estimate  of  how  well  this  student 

will  do  on  his  next  training  flight.  Place  an  "X"  squarely 
within  one  category. 


Unac-  Marginally  Below  Average  Above  Very  Outstand 
ceptable  Acceptable  Average  Average  Good  ing 


*  T-39  Flights  Only 
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APPENDIX  E 

CONTENT  OF  DEVICE  1D23  EVALUATION 
STUDENT  QUESTIONNAIRE 
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The  follov;ing  questionnaire  items  are  intended  to  allow 
you  -  the  student  -  to  critique  various  design  features  of 
Device  1D23.  Your  comments  will  be  of  considerable  value  to 
the  Chief  of  Naval  Training  by  providing  guidelines  regarding 
desirable  design  features  (which  can  be  included  in  future 
training  devices)  and  undesirable  design  features  (which  should 
be  avoided  in  future  training  devices) . 

Your  comments  on  this  questionnaire  will  remain  anonymous. 
Please  feel  free  to  express  your  viewpoints  openly. 

Please  feel  free  to  write  in  additional  comments 
wherever  you  feel  they  are  appropriate. 

1.  What  is  the  most  advanced  Training  Problem  (TP) 
on  which  you  have  received  instruction?  _ 


2.  What  problems,  if  any,  have  you  experienced  in  using 
the  communication  link  (ICS)  between  yourself  and 
the  instructor? 

3.  What  problems,  if  any,  have  you  experienced  in 
using  the  communication  link  (ICS)  between  yourself 
and  the  TDO? 

4.  Using  the  following  rating  scale,  place  an  "X"  in 
the  category  which  best  reflects  your  experience 
regarding  the  value  of  the  computer  printout  of 
student  performance  for  reconstructing  the  flight. 


1 

_ ) 

No  Minimal 

V  lue  Value 

5.  Using  the  following  rati 
the  category  which  best 
regarding  the  value  of  t 
student  performance  for 

Moderate  High 

Value  Value 

ng  scale,  place  an  "X*  in 
reflects  your  experience 
he  computer  printout  of 
evaluating  how  well  you  did. 

No  Minimal  Moderate  High 

Value  Value  Value  Value 


5.  From  a  student's  standpoint,  what  are  three  of  the 
most  important  improvements  which  could  be  made  to 
the  design  of  Device  1D23? 

7  From  a  student's  standpoint,  what  are  three  of  the 
most  desirable  design  features  which  are  incorporated 
in  Device  1D23? 
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8.  This  item  addresses  the  period  when  you  were  first 
introduced  to  Device  1D23  and  began  to  receive 
training  in  the  device  (e.g.  TPs  1-4).  Using  the 
rating  scale  below,  place  "XsM  in  the  appropriate 
columns  to  indicate  your  assessment  of  the  degrees 
of  difficulty  v/hich  you  expel ienced  in  each  of  the 
areas  listed. 


Identifying  uses  of  each 
cockpit  functional  area 


Interpreting  digital 
readout  displays 


Using  keyboards  for 
data  inputs 


Learning  requirements  and 
procedures  for  control  of 
the  device 


Communicating  with  TDOs 


Communicating  with  INFOs 


Levels  of  Difficulty 


9.  Please  describe  any  other  areas  where  you  have 
experienced  difficulty  because  of  the  way  Device 
1D23  was  designed. 

10.  Please  feel  free  to  make  any  additional  comments  about 
the  design  of  Device  1D23. 
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